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FOREWORD 


As  the  Bureau  of  Quality  Assurance  began  implementation  of  the  PSRO  program 
in  hospitals,  it  soon  became  evident  that  the  future  would  require  PSROs  to 
be  actively  involved  in  ambulatory  medical  care  quality  assurance.  Since 
the  state  of  the  art  of  ambulatory  care  review  is  far  less  advanced  than 
that  of  hospital  review,  we  realized  that  it  would  be  necessary  to  develop 
a  sound  basis  for  the  careful,  gradual  movement  of  PSROs  into  ambulatory 
care  quality  assurance. 

This  three-volume  report  is  the  result  of  the  work  completed  by  Health  Care 
Management  Systems,  Inc.,  in  one  of  the  first  major  Bureau  contracts  to 
acquire  and  document  knowledge  of  presently  operational  ambulatory  care 
quality  assurance  projects.    The  purpose  of  this  project  was  to  systematically 
assess  and  document  existing  activities  in  a  variety  of  health  delivery  set- 
tings across  the  country.    During  the  course  of  the  project,  a  generic 
quality  assurance  model  was  developed  for  use  in  the  design,  implementation, 
and  assessment  of  ambulatory  care  review  systems.    This  model  is  now  under 
refinement  and  is  being  applied  in  the  Bureau's  cooperative  ambulatory  care 
quality  assurance  demonstration  project.    It  represents  a  significant 
advance  in  our  ability  to  design  and  assess  such  systems. 

This  three-part  report,  which  includes  a  description  of  the  generic  model 
and  study  findings,  description  of  each  of  the  27  ambulatory  quality 
assurance  systems  included  in  the  survey,  and  a  bibliographic  index  that 
has  been  enriched  by  selected  abstracts,  should  provide  the  reader  with  a 
concise  overview  of  the  state  of  the  art.    The  results  of  this  project  are 
an  important  part  of  the  foundation  upon  which  to  build  well-balanced  and 
meaningful  PSRO  participation  in  ambulatory  care  quality  assurance.  The 
report  also  reflects  the  investigators'  concern  with  ambulatory  quality 
assurance  outside  the  scope  of  PSRO. 

The  authors  of  the  report  developed  their  material  in  a  relatively  short 
time  to  meet  the  need  for  fundamental  knowledge  of  existing  quality 
assurance  programs  to  support  the  implementation  of  voluntary  systems  and 
governmental  regulatory  programs.    It  is  hoped  that  any  limitations  in 
the  material  presented  herein  will  be  corrected  through  additional  research 
and  careful  experimentation  and  demonstration  of  new  ambulatory  quality 
assurance  methods  and  organization. 

We  believe  that  this  report  will  be  valuable  to  those  within  and  outside 
the  Federal  Government  who  are  interested  in  establishing  effective  quality 
assurance  systems  in  ambulatory  medical  care.    We  are  pleased  to  have 
provided  partial  funding  to  the  project  and  the  preparation  of  this  mono- 
graph and  hope  that  it  will  be  useful  to  all  who  are  now  engaged  in 
activities  leading  to  improved  quality  in  ambulatory  medical  care. 


Michael  J.  Goran,  M.D. 
Director 

Bureau  of  Quality  Assurance 
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ABSTRACT 


That  quality  care  assurance  is  in  a  state  of  flux  is  attested  to  by 
current  literature  in  the  field  and  is  substantiated  by  the  Ambulatory 
Care  Qual ity  Assurance  Project  (ACQAP)  report.    This  nationwide  survey 
was  undertaken  to  reveal  the  state  of  the  art  in  the  United  States 
today.    During  a  study  of  more  than  two  dozen  sites  over  a  wide  range 
of  geographic  locations,  the  ACQAP  produced  a  set  of  basic  program 
procedures,  a  scoring  method,  and  what  appears  to  be  a  sound  basis 
for  planning,  operating  and  assessing  quality  care  assurance  programs. 
The  literature  relative  to  quality  assurance  in  ambulatory  care  was 
investigated  and  a  collection  of  the  better  references  accumulated. 

Volume  I  of  the  three-volume  report  contains  a  detailed  description 
of  the  ACQAP  Model,  its  scoring  system,  and  its  findings.    The  data 
collection  instrument,  a  226-item  open-ended  questionaire  used  in 
the  study,  is  contained  in  an  appendix. 

Volume  II  contains  detailed  descriptions  of  the  27  programs  reviewed 
in  the  course  of  the  ACQAP  accompanied  by  a  compact  tabulation  of 
their  key  characteristics  for  quick  reference,  so  that  the  reader 
may  compare  the  study  sites  with  his  own. 

Volume  III  is  a  bibliography  containing  a  general  list  of  the  articles 
and  other  references  found  to  be  most  useful  to  the  study.    The  list 
is  duplicated  under  several  topic  headings  for  the  convenience  of  the 
reader  interested  in  a  particular  aspect  of  quality  assurance  in 
ambulatory  medical  care.    A  number  of  the  references  have  been 
abstracted  and  the  volume  also  contains  a  narrative  review  of  the 
1 iterature. 
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PROJECT  BACKGROUND 


Today's  medical  and  political  literature  reflects  concern  over  the 
definition,  feasibility,  acceptability,  and  cost  of  conducting  widespread 
review  of  the  quality  of  medical  services.    Review  of  hospital  services 
already  has  received  much  attention.    Regarding  the  activity  of  Professional 
Standards  Review  Organizations  (PSRO),  Hellman  (1976)  states,  "By  the 
end  of  F.Y.  1977,  120  conditional  PSROs  will  be  performing  review  of  about 
three  million  hospital  admissions."    Progress  in  the  review  of  services  to 
ambulatory  patients  has  been  significantly  slower  and  less  documentation 
on  the  extent  of  activity  is  available.    This  report  is  the  result  of 
interest  by  the  Bureau  of  Quality  Assurance,  Department  of  Health,  Education 
and  Welfare,  in  expanding  the  information  base  regarding  the  state  of  the 
art  in  ambulatory  quality  assurance  activity. 

Through  funding  from  The  Robert  Wood  Johnson  Foundation,  Health  Care 
Management  Systems,  Inc.  (HCMS)  is  engaged  in  a  three-year  project,  part 
of  which  is  intended  to  expand  knowledge  of  ambulatory  care  quality 
assurance  and  to  improve  methods  for  assessing  quality  of  ambulatory 
medical  services.    This  report  also  reflects  concern  with  ambulatory  quality 
assurance  extending  outside  the  scope  of  PSRO. 

According  to  Tenney  (1972),  ambulatory  medical  care  is  a  broad  term 
usually  defined  as  health  services  rendered  to  patients  under  their  own 
cognizance,  at  a  time  when  they  are  not  in  a  hospital  or  other  health  care 
institution.    The  National  Center  for  Health  Services  Research  and  Develop- 
ment (1972)  defines  an  'encounter'  as  a  "face-to-face  contact  between  a 
patient  and  a  provider  who,  at  the  time  of  contact,  has  primary  responsi- 
bility for  assessing  and  treating  or  managing  the  condition  of  the  patient 
and  who  exercises  independent  judgment  as  to  the  care  of  the  patient." 

Ambulatory  medical  care  is  the  largest  component  of  the  he'alth  care 
delivery  system  in  this  country.    In  1972,  the  National  Center  for  Health 
Statistics  estimated  that  a  billion  physician  visits  (including  telephone 
contacts)  were  made  in  the  United  States,  an  average  of  5.0  per  person. 
Seventy-three  percent  of  the  population  consulted  a  physician  at  least 
once  that  year.    In  contrast,  there  were  28.5  million  discharges  from  short- 
stay  hospitals  in  1972,  an  average  of  0.14  per  person. 

Definition  of  the  concept  of  "quality  assurance"  appears  to  vary 
with  the  intent  of  the  user.    In  a  review  of  quality  of  care  research, 
Shortridge  (1974)  reported  ten  different  verbal  and  operational  definitions 
for  ten  research  projects  reviewed.    Brook  (1975)  makes  the  distinction 
between  quality  assessment  and  quality  assurance:    quality  assessment  means 
measuring  the  level  of  quality  provided  at  some  point  in  time,  but  connotes 
no  effort  to  change  or  improve  that  level  of  care,  whereas  quality  assurance 
means  both  measuring  the  level  of  care  provided  and,  when  necessary,  improving 
it.    The  term  quality  assurance  is  used  accordingly  in  this  report. 


1 


Public  interest  in  quality  assurance  review  is  evidenced  by  the  passage 
of  Public  Law  92-603  which  mandates  PSRO  review  operations.    Section  1155 
(a)  (1)  states  that  it  is  the  duty  and  function  of  each  PSRO  to  assure,  at 
the  earliest  date  practicable,  responsibility  for  the  review  of  professional 
activities  of  physicians  and  other  health  care  practitioners  and  institutional 
and  noninsti tutional  providers  of  health  care  services  to  determine  whether 
such  services  and  items  are  medically  necessary  and  whether  the  quality  of 
such  services  meets  professionally  recognized  standards  of  health  care. 

Although  ambulatory  care  review  is  optional  for  PSRO,  the  mandate  clearly 
states  the  desirability  of  establishing  such  review  at  the  earliest  date 
possible.    It  becomes  apparent  that  in  order  for  regulatory  interests  to 
establish  sound  technical  guidelines  for  the  operation  of  ambulatory  quality 
review  programs,  examination  of  technical  issues  is  required. 

To  the  degree  to  which  other  interests,  including  physicians  in 
individual  and  group  practice,  state  and  local  government  agencies, 
accreditation  organizations,  and  the  general  public  are  concerned  with 
the  cost  and  quality  of  medical  care,  technical  issues  are  important 
outside  PSRO  as  well. 

The  Ambulatory  Care  Quality  Assurance  Project  represents  the  first 
phase  of  a  two-phase  research  effort  aimed  at  setting  priorities  for 
ambulatory  care  quality  assurance  review  by  the  Bureau  of  Quality  Assurance. 
Figure  1,  "Bureau  of  Quality  Assurance  Plan:    Ambulatory  Medical  Care 
Review,"  presents  the  Phase  I  and  II  objectives.    Therefore,  the  purpose 
of  this  investigation  was  to  begin  to  identify  and  set  priorities  for 
areas  of  ambulatory  care  quality  assurance  that  require  future  examination. 
After  the  first  year  survey,  reported  here,  work  will  take  the  form  of 
a  series  of  demonstration  and  experimental  projects  designed  to  investigate 
technical  issues  in  a  more  definitive  manner. 

The  general  goal  of  Phase  I  of  the  project  was  to  describe  past  and 
currently  used  methods  of  ambulatory  care  quality  assurance  review  and  to 
develop  major  models  and  strategies  for  ambulatory  care  quality  assurance 
evaluation.    The  general  goal  was  further  defined  to  correspond  with  (a)  the 
more  specific  needs  of  the  Bureau  of  Quality  Assurance  to  identify  technical 
issues  which  must  be  examined  before  PSRO  ambulatory  review  regulatory 
action  is  initiated  and  (b)  the  research  needs  as  reflected  in  current 
literature  on  the  topic  of  ambulatory  care  quality  assurance  review.  More 
specifically,  the  objectives  of  the  project  were: 

-  To  describe  the  current  state  of  the  art 

-  To  develop  a  classification  system  for  quality  assurance  programs 
implemented  in  various  types  of  delivery  systems 

-  To  design  initial  methods  for  assessing  quality  assurance  programs 

-  To  identify  technical  design  and  implementation  strengths;  to 
place  priority  on  weaknesses  which  should  receive  attention  in 
future  research 
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FIGURE  1 


BUREAU  OF  QUALITY  ASSURANCE  PLAN:    AMBULATORY  MEDICAL  CARE  REVIEW 

Goal 


To  implement  effective  and  acceptable  ambulatory  medical 
care  quality  assurance  review  mechanisms  in  PSROs. 


Research  and  Demonstration  Objectives 

Phase  I  (1975-1976) 

Objective: 

To  identify  technical  problems  that  require  further 
investigation. 

Method: 

Survey  quality  assurance  programs  throughout  the 
United  States. 

Apply  assessment  tools;  identify  and  set  priorities 
for  problems  that  require  investigation;  specify  the 
issues  to  be  tested  in  Phase  II. 

Result: 

Recommendations  to  the  Bureau  of  Quality  Assurance 
setting  priorities  for  the  types  of  problems  that 
must  be  investigated  during  the  Phase  II  demonstration 
projects. 

Phase  II  (1976-1978) 

Objective: 

To  investigate  technical  problems  identified  in 
Phase  I. 

Method: 

Implement  PSRO  demonstration  projects. 

Conduct  specialized  studies  to  investigate  problems. 

Result: 

Recommendations  to  the  Bureau  of  Quality  Assurance 
for  administrative  and  regulatory  consideration. 
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In  the  course  of  the  project  a  model  for  the  measurement  of  programs 
that  attempt  to  assure  high  quality  in  the  medical  care  of  ambulatory 
patients  was  developed.    Described  in  this  volume  is  the  model  itself  and 
how  it  was  applied  to  a  survey  of  quality  assurance  programs. 

The  project  included  on-site  review  of  27  quality  assurance  programs 
in  locations  throughout  the  United  States.    Volume  II,  "Program  Descriptions" 
presents  detailed  descriptions  of  the  sites  visited  and  their  programs. 

Volume  III,  "Bibliography  and  Selected  Abstracts,"  contains  an  index, 
culled  from  a  large  selection,  of  journal  articles  and  other  references 
which  are  notable  for  their  relevance,  timeliness  and  general  excellence. 
The  volume  also  contains  subindexes  of  references  from  the  main  index, 
reclassified  by  topic.    Selected  articles  are  abstracted  and  a  review  of 
the  literature  is  presented. 
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SURVEY  DESIGN 


The  design  used  in  this  study  was  intended  to  be  of  a  descriptive 
type,  as  opposed  to  an  experimental  study.    A  rigorous  descriptive  study 
is  required  to  have  breadth  and  depth  and  must  insure  that  the  data  collected 
are  reliable  and  valid.    Questionnaires  and  other  data  collection  instruments 
must  be  designed  and  thoroughly  tested  with  a  view  to  avoiding  error  and 
bias.    They  must  be  used  by  well  trained  and  qualified  persons  and  should 
elicit  a  sufficiently  large  sample  of  needed  data  to  accurately  represent 
the  population  being  studied  (Galfo  and  Miller,  1965). 

Findings  of  this  study  are  offered  in  the  light  of  the  requirements 
outlined  above.    Because  of  the  sampling  design  dictated  by  the  nature 
of  the  investigation,  no  specific  generalizations  to  populations  outside 
the  study  sample  were  intended.    However,  all  findings  may  be  seen  as  hypotheses 
for  future  study  and  empirical  refinement.    Details  of  methods  used  are 
presented  in  Appendix  I,  "Ambulatory  Care  Quality  Assurance  Program  Project 
Methods." 

Figure  2,  "Survey  Design,"  illustrates  the  conceptual  plans  for  the 
survey.    Under  contract  stipulation,  the  survey  was  to  include  an  unspecified 
number  of  ambulatory  care  quality  assurance  programs  throughout  the  United 
States.    The  27  sites  finally  included  in  the  study  were  selected  on  the 
basis  of  the  following  screening  criteria: 

(1)  Evidence  of  ambulatory  quality  assurance  program  in  operation 

(2)  Ability  to  provide  initial  documentation  regarding  quality  assurance 
(Q.A. )  activities 

(3)  Evidence  of  high  interest  and  willingness  to  participate  in  the 
time  consuming  site  visit  and  data  collection  procedures 

(4)  Location  within  the  continental  boundaries  of  the  U.S. 

(5)  Capability  of  being  classified  by  the  majority  of  selected  analysis 
factors 

The  sites  included  in  this  survey  were  selected  using  what  Riley  (1963) 
calls  the  focused  sample.    A  focused  sample  allows  the  researcher  to  concen- 
trate on  the  explanatory  variables  of  immediate  research  interest  by  holding 
constant  other  extraneous  variables  outside  his  concern.    Extraneous  variables 
are  controlled  through  the  sampling  procedure  itself  rather  than  by  statistical 
analysis. 

A  sample  becomes  less  representative  when  it  is  selected  using  pre- 
determined criteria,  but  it  does  provide  a  means  for  assessing  and  evaluating 
the  relationship  among  variables.    In  this  study,  the  criteria  for  site 
selection  may  bias  the  study  sample.    Aside  from  including  only  fully 
operational  quality  assurance  programs  which  could  provide  program  documenta- 
tion, the  voluntary  nature  of  the  sample  may  have  introduced  an  additional 
bias.    Use  of  a  focused  sample  coincides  with  the  purpose  of  this  investi- 
gation, which  is  to  survey  and  assess  operational  programs.    The  focused 
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FIGURE  2 
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Q.A.  -  Quality  Assurance 


sample  helps  to  eliminate  the  unwanted  and  opposite  bias  of  including 
ineffective,  poorly  organized,  undocumented  and  unsystematic  quality  assurance 
programs  in  the  study. 

The  survey  group  was  classified  by  the  factors  listed  in  Figure  2. 
Definitions  of  the  factors  and  results  of  the  classification  are  presented 
in  Figure  3.    If,  in  certain  cases,  a  site  could  not  be  classified  because 
of  unavailable  data,  or  a  factor  that  was  not  suitable,  the  site  was  excluded 
from  classification  and  analysis  for  that  particular  factor.  Further 
details  of  the  site  identification,  selection,  and  classification  procedures 
are  included  in  Appendix  I,  "Ambulatory  Care  Quality  Assurance  Project 
Methods." 

The  Data  Collection  Instrument,  developed  and  pilot  tested  by  survey 
staff,  consisted  of  a  37-page,  226-item,  open  end  questionnaire  divided 
into  two  major  sections:    the  Delivery  System  and  the  Quality  Review 
Program.    Figure  2  lists  the  content  areas  at  which  the  questionnaire 
was  aimed.    The  questions  are  listed  in  Appendix  II  of  this  volume.  Data 
collection  for  each  quality  assurance  program  involved  previsit,  on-site 
visit,  and  postvisit  information  gathering  to  complete  the  items  in  the 
Data  Collection  Instrument.    In  order  to  assess  systematically  the  diverse 
quality  assurance  programs,  it  was  necessary  to  develop  an  assessment 
tool  which  would  provide  a  framework  for  site  analysis.    The  development 
and  application  of  the  final  assessment  instrument,  called  the  Ambulatory 
Care  Quality  Assurance  Program  (ACQAP)  Model,  is  described  in  this  volume. 

Data  analysis  was  aimed  at  two  sets  of  questions.    The  primary  questions 
this  study  addresses  focus  on  issues  related  to  the  technical  content  of 
ambulatory  care  quality  assurance  programs  (Factor  No.  0).    Secondary  survey 
questions  are  aimed  at  the  relationship  between  scores  and  delivery  setting 
and  quality  assurance  program  (Factors  1-6)  characteristics. 

The  survey  questions  are  outlined  below: 

Primary  Survey  Questions  (Factor  No.  0,  Technical  Q.A.  Activity): 

1.  In  the  ACQAP  Model,  what  is  the  relationship  between  planning 
and  operation  scores? 

2.  How  do  ACQAP  Model  scores  rank  across  all  survey  sites? 

3.  Which  components  in  the  ACQAP  Model  scored  lowest?  Highest? 

4.  Which  procedures  in  the  ACQAP  Model  scored  lowest?  Highest? 

5.  What  informal  findings  might  be  hypothesized  for  future  testing 
and  development? 
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FIGURE  3 

SURVEY  SITES  CLASSIFIED  BY  FACTORS 


(N=27) 


Factor 
Number  and  Categories 

Operational  Definitions  * 

No.  Sites 
Included  In 
Analysi  s 

Total 
Sites 
Analyzed 

(0) 

Technical  Q. A. ***  Activity 

Related  to  ambulatory  qualtiy  assurance  subject  matter  as  reflected  in 
current  literature;  includes  activities  outlined  in  the  ACQAP  Model.  *** 

Q.A.  Program 

Evidence  of  actual  or  intended  quality  assurance  activity  in  all  nine 
components  of  the  ACQAP  Model 

23** 

Q.A.  Research 

Conducts  in  depth  study  of  specified  components  of  a  review  system;  no 
intended  ongoing  activity  in  all  nine  components  of  the  ACQAP  Model . 

Not 
i  n 
See 

3 

incl uded 
analysis 
Vol.  II 

23 

(1) 

Type  of  Group  Practice 

Delivery  of  medical  services  by  3  or  more  physicians  formally  organized 
to  provide  medical  call,  consultation,  diagnosis,  and  treatment;  joint  use 
of  equipment  in  one  facility;  income  distributed  in  accordance  with  methods 
usually  determined  by  members  of  the  group. 

^"innlp  Snprifllt"\/ 
o  i  ny  i  c    jjjct  i  a  i 

Composed  exclusively^  physicians  in^  specialty  other  than  General  Practice 

3 

General  Practice 

Composed  exclusively  of  general  practitioners. 

4  ! 

17 

Multi-Specialty 

Composed  of  physicians  in  2  or  more  specialties 

10 

(2) 

Type  of  Revenue  Source  to 
Delivery  System 

Fee  for  Service 

For  at  least  80%  of  service  rendered,  fee  schedule  is  calculated  from  which 
patients  are  billed  charaes. 

3 

Prepaid 

For  at  least  80%  of  services  rendered,  a  guaranty  of  services  is  offered  by  a 
plan  to  an  enrolled  group  in  exchange  for  a  prepaid  per  capita  fee  or  involves 
a  quaranty  of  services  supported  through  prepaid  governmental  grant  or  funds. 

i 

18 

Both  FFS  and  PP 

A  combination  of  billing  patients  based  on  schedule  and  guaranty  of  service 
.based  on  prepaid  patient  fees 

 1 

4 

(3) 

Length  of  Q.A.  Program 
Acti  vi  ty 

0-24  Months 

Formally  allocated  resources  or  initiated  activity  in  the  system  surveyed  by 
the  ACRE  project  within  24  months  before  June,  1976 

12  [ 

25  Months  and  Over 

Formally  allocated  resources  or  initiated  activity  in  the  system  surveyed  by 
the  ACRE  project  25  or  more  months  pervious  to  June,  1976 

11 

23 

(  A  \ 
(4) 

Type  of  Q.A.  Program  Support 

Resources  allocated  to  develop  and  operational  i r.e  a  quality  assurance  program 

Vol unteer 

Resources  allocated  are  not  budgeted  for  specific  activities  covered  in  a 
quality  assurance  program,  but  are  internalized  in  personnel  salaries  or 
facility  overhead. 

10 

i 

Budgeted 

Resources  allocated  are  budgeted  line  item  funds  from  within  the  organization 
or  are  derived  from  sources  outside  the  organization  (gov't  or  private 
agencies)  and  are  budgeted  specifically  for  quality  assurance  activities. 

13 

23 

(5) 

Type  of  Q.A.  Review 

  j 

Internal 

Review  conducted  by  in-house  affiliated  group. 

17 

22 

External 

Review  conducted  by  outside  nonaffiliated  group. 

5 

(6) 

Type  of  Data  Source  for  Q.A. 
Program 

Claim  Form 

A  document  with  a  series  of  data  elements  used  to  denote  medical  procedures 
and  activities;  submitted  for  payment  or  reimbursement  of  cost  of  those  medical 
activities.    May  be  submitted  to  hospital  insurance  companies,  agencies  em- 
powered to  distribute  Title  V,  XVII  and  XIX  funds,  or  other  fiscal  intermed- 
iaries. 

3 

Chronological  Medical  Record 

Data  in  the  record  are  not  organized  by  problem.    Information  may  be  organ- 
ized in  chronological  sequence  as  it  is  recorded  or  returned  to  the  record 
or  in  other  convenient  ways. 

7 

i 

19 

1 

Problem  Oriented  Medical 
Record 

A  record  system  which  emphasizes  a  list  of  patient  problems  to  be  managed,  in- 
cluding preventive,  psychiatric  and  environmental  problems.    The  plans  for 
each  problem,  progress  notes  and  appraisal  are  related  to  the  problem  list, 
which  is  kept  current  on  a  face  sheet  on  the  patient's  chart.    The  record 
follows  a  specific  format  through  S.O.A.P.  (****  which  asks  for  subjective 
assessment,  objective  assessment  and  plans  and  analyses. 

9 

Sources  of  definitions  are:    AMA  Classification  of  Practices  and  Financing  Mechanisms,  Review  of  Administrative  Medical 
Literature,  and  various  definitions  used  by  government  agencies  and  fiscal  intermediaries. 

**     Of  the  27  Survey  Sites,  4  were  not  included  in  this  analysis  since  3  were  research  projects  and  1  had  missing  data  at  the 
time  of  analysis. 

Q.A.  -  Quality  Assurance 

*+** 

S.O.A.P.  -  Subjective  -  Objective  Assessment  Plan 


Secondary  Survey  Questions:    (Factors  1  to  6,  Setting  and  Q.A.  Program 
characteristics) : 

1.    What  factors  and  categories  appear  to  affect  the  ACQAP  Model  total 
score? 

a.  Type  of  Group  Practice  (single  specialty,  general  practice, 
and  multi special ty) 

b.  Type  of  Revenue  Source  to  Delivery  System  (fee  for  service, 
prepaid,  and  both  fee  for  service  and  prepaid) 

c.  Length  of  Quality  Assurance  Program  Activity  (0  to  24  months 
and  25  months  and  over) 

d.  Type  of  Quality  Assurance  Program  Support  (volunteer  and 
budgeted) 

e.  Type  of  Quality  Assurance  Review  (internal  and  external) 

f.  Type  of  Data  Source  for  Quality  Assurance  Program  (claim 
form,  chronological  medical  record,  and  problem-oriented 
medical  record) 
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THE  AMBULATORY  CARE  QUALITY  ASSURANCE  PROJECT  MODEL 


This  section  presents  the  rationale  behind  construction  of  an  ambulatory 
care  quality  assurance  model.    Both  widespread  needs  for  research  and  the 
assessment  requirements  of  the  Ambulatory  Care  Quality  Assurance  Project  (ACQAP) 
are  discussed;  the  purpose  of  the  ACQAP  Model  is  specified. 


STATEMENT  OF  PURPOSE 

Ambulatory  care  quality  assurance  research  and  development  has  been 
aimed  at  select,  and  oftentimes  isolated,  program  elements.    For  example, 
extensive  effort  has  been  applied  to  such  areas  as  program  identification 
and  criteria  development  (Gonnella  et  al.,  1975,  1976;  Hare  and  Barnoon, 
1973;  Kessner  et  al . ,  1973;  Novick  et  al.,  1976;  Rosenburg,  1975;  Thompson 
and  Osborne,  1974,  1976;  Wagner  et  al . ,  1976;  Williamson,  1971,  1973); 
data  sources  (Fessel  and  Van  Brunt,  1972;  Hurst,  1971;  Komaroff  et  al.,  1973; 
Kaufman  et  al.,  1974;  Lyons  and  Payne,  1974;  Margolis  et  al . ,  1973;  Thompson 
and  Osborne,  1976;  Weed,  1971;  Wirtschafter  and  Mesel ,  1976);  and  corrective 
action  mechanisms  (Brook  et  al. ,  1975;  Brown  and  Uhl ,  1970;  Pozen  and 
Bonnet,  1976;  Slee,  1972). 

The  emphasis  of  research  on  selected  areas  of  a  quality  assurance 
program  is  commendable  since  valid  methods  and  approaches  must  continue 
to  be  conceptualized,  tested,  and  described  before  widespread  use  should 
be  recommended.    It  is  this  type  of  research  which  must  continue  to  ask 
the  question,  "How  val id  is  the  problem  identification,  criteria,  and  review 
process?" 

The  ACQAP  Model  addresses  a  broader  question:    "How  is  an  ambulatory 
care  quality  assurance  program  defined?    What  are  the  minimum  elements, 
procedures,  and  requirements  of  a  comprehensive  program?"    It  should  be 
noted  that  the  first  question  regarding  the  validity  of  procedures  is  a 
subset  of  the  second  question  regarding  program  requirements.    That  is, 
if  one  develops  minimum  program  requirements,  establishing  the  validity  of 
methods  is  one  requirement  of  primary  importance.    The  ACQAP  Model  represents 
an  attempt  to  consolidate  what  has  been  learned  from  specific  research 
areas,  programs  and  approaches  into  a  list  of  specifications  applicable 
to  all  ambulatory  review  systems. 

Currently,  the  literature*  is  devoid  of  formalized  tools  aimed  at 
specifying  minimum  requirements  of  a  comprehensive  ambulatory  care  quality 
assurance  program.    There  are  publications  indirectly  related  to  requirements 
which  appear  in  the  form  of  procedural  or  program  manuals.    Program  manuals 
for  inpatient  review  specify  requirements  and  procedures  for  individual 
approaches  (American  Hospital  Association,  1972;  Department  of  Health, 
Education  and  Welfare,  1975;  Joint  Commission  on  Accreditation  of  Hospitals, 
1973). 


*  See  Volume  III,  Ambulatory  Care  Quality  Assurance  Project:  Bibliography 
and  Selected  Abstracts  of  this  report. 
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Recently,  reports  describing  individual  ambulatory  review  approaches 
have  begun  to  emerge  (Michnich  et  al.,  1976;  Rubin,  1973;  Williamson, 
1973)  but  the  literature  for  the  most  part  focuses  on  specific  methods 
for  developing  and  conducting  review  rather  than  on  requirements  for  programs 
in  general . 

In  addition  to  needing  the  ACQAP  Model  to  complement  conceptualization 
and  definition  of  ambulatory  quality  review  systems,  HCMS  investigators, 
while  designing  the  project,  required  an  evaluative  tool  which  could  be 
used  to  describe,  distinguish,  and  assess  elements  of  ambulatory  quality 
assurance  programs  throughout  the  United  States.    This  research  need  was 
satisfied  through  subsequent  ACQAP  Model  development.    In  summary,  the 
purpose  for  development  of  an  ACQAP  Model  was:    1)  To  begin  to  operationally 
define  minimum  requirements  of  ambulatory  quality  assurance  programs;  and 
2)  To  have  an  evaluative  tool  for  program  assessment. 

The  following  material  explains  the  Ambulatory  Care  Quality  Assurance 
Project  (ACQAP)  Model  as  it  was  developed  for  the  project.    The  developmental 
procedures,  review  by  consultants,  and  resulting  refinements  are  discussed. 
Finally,  model  limitations  and  future  plans  are  described. 


THE  ACQAP  MODEL 

The  model  presented  in  Figure  4,  "Components  of  Ambulatory  Care  Quality 
Assurance  Model,"  consists  of  nine  components.    Each  component  has  a  "goal" 
describing  the  purpose  of  activity.    Within  each  component  (except  Component 
I)  are  "Planning  Procedures"  and  "Operational  Procedures"  designed  to  specify 
all  managerial  and  methodological  tasks  which  must  be  achieved  if  the 
component  goal  is  to  be  met.    The  model  consists  of  99  broadly  defined 
procedures:    39  planning  and  60  operational. 


MODEL  DEVELOPMENT 

Investigators  designed  the  model  according  to  three  basic  criteria. 
The  model  must: 

1.  Represent  minimum  requirements  (procedures)  of  any  formal 
quality  assurance  program  independent  of  the  method  applied 

2.  Outline  procedures  necessary  for  systematic  and  thorough 
program  design  and  implementation 

3.  Lend  itself  to  further  refinement,  particularly  to  the 
development  of  indicators  which  describe  the  degree  to 
which  requirements  (procedures)  have  been  met 

Initial  procedures  outlined  in  the  model  were  identified  and  consoli- 
dated through  a  review  of  literature*  on  ambulatory  care  quality  assurance 
research  and  operational  programs  and  through  on-site  orientation  to  opera- 
tional quality  assurance  programs  by  HCMS  investigators. 


*  See  Volume  III,  Ambulatory  Care  Quality  Assurance  Project:  Bibliography 
and  Selected  Abstracts,  of  this  report. 
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To  help  assure  the  validity  and  appropriateness  of  model  components, 
goals,  and  procedures,  the  first  draft  of  the  ACQAP  Model  was  distributed 
to  15  quality  assurance  consultants  in  related  fields  throughout  the 
country.    The  consultants,  listed  in  Appendix  II,  were  asked  to  critically 
comment  on  the   model  content  and  to  make  additions  or  deletions.  Once 
input  was  received  comments  were  summarized  in  tabular  form  to  assure 
anonymity.    The  study  staff  then  reviewed  consultant  comments  with  two 
advisory  board  members,  and  modifications  were  incorporated  into  the  final 
model . 


SCORING  METHODS 

The  survey  of  ambulatory  care  quality  assurance  review  systems  required 
ACQAP  Model  scoring  methods  that  would  indicate  the  degree  to  which  each 
procedure  had  been  accomplished  for  each  system  reviewed.    Figure  5,  "Scoring 
Criteria,"  presents  a  description  of  the  three-point  (0,1,2)  scoring  system 
for  both  planning  and  operational  procedures  contained  in  the  model. 

To  control  bias  in  the  assignment  of  scores,  the  scoring  process 
was  divided  into  two  stages. 

1.  Independent  Scoring  Session 

Two  or  three  HCMS  staff  members,  each  working  independently, 
scored  the  level  of  activity  for  each  site  on  all  objectives. 
This  process  involved  three  steps: 

a.  Obtaining  the  completed  reference  file  and  reviewing 
all  information  in  the  file 

b.  Assigning  appropriate  score  values  to  each  objective 
for  all  components 

c.  Coding  these  results  on  the  prescribed  tally  sheet 

2.  Joint  Assessment  Meeting 

To  assign  a  final  score  to  each  site,  the  raters  met  to  resolve 
disagreements  and  come  to  a  consensus.    The  assessment  meetings 
usually  involved  extensive  discussion,  justification  and  reexamina- 
tion of  information  to  support  the  assignment  of  a  particular  score. 
A  single  consensus  score  was  assigned  to  each  procedure. 

Interrater  reliabilities  were  calculated  for  the  two  or  three  raters 
assigned  to  score  each  site.    Reliabilities  were  determined  by  subtracting 
the  ratio  (expressed  as  a  percentage)  of  the  number  of  disagreements 
between  judges  to  the  total  possible  disagreements  from  1.00.  Reliabilities 
between  judges  ranged  from   a  low  of  0.68  to  a  high  of  0.99  with  an  average 
overall  reliability  of  approximately  0.87.    Appendix  IV,  "Interrater 
Reliabilities,"  presents  the  reliability  scores  for  each  site. 
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FIGURE  5 
Scoring  Criteria 


Points  Planning  Procedure  Criteria 


0      No  plan  for  the  procedure  has  been  made.    No  documents  address  the 
need  for  the  procedure,  no  mention  made  during  the  site  visit  that 
a  systematic  effort  had  been  made  to  discuss  the  procedure  with 
other  members  of  the  site  staff. 


1       The  procedure  has  been  partially  planned  or  is  in  the  planning 
process.    This  means  that  the  procedure  is  not  operational  and  is 
not  ready  for  implementation.    This  partially  planned  procedure 
may  be  documented  in  written  or  verbal  form,  but  there  must  be 
evidence  that  some  systematic  attempt  has  been  made  to  discuss 
and  consider  the  procedure  with  other  staff  members. 


2  The  procedure  is  fully  planned  and  ready  for  implementation. 
The  procedure  must  be  documented  in  written  form  by  the  site 
staff. 


Points  Operational  Procedure  Criteria 


0      The  procedure  is  not  implemented  or  operational.    There  is  no 
written  or  verbal  evidence  that  the  procedure  is  a  part  of  the 
quality  assurance  program. 


1       Procedure  is  partially  implemented  or  operational.    This  means 

that  the  procedure  is  not  administered  in  a  systematic  and  routine 
way.    The  procedure  may  be  addressed  in  an  informal  and  irregular 
fashion. 


2      The  procedure  is  fully  operational.    This  means  that  the 
procedure  is  a  routine  and  systematic  part  of  the  quality 
assurance  program.    Explicit  schedules  and  time  frame  dictate 
the  operation  of  the  procedure. 
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To  standardize  scores  for  all  components,  the  ratios  of  achieved 
scores  (0,1,2)  to  total  possible  scores  were  calculated.    For  example  if 
a  component  had  six  procedures,  then  a  total  possible  score  would  be  12 
(i.e.,  2x6).    If  the  component  received  a  total  of- only  six  points 
(e.g.,  one  for  each  procedure),  then  the  achieved  score  would  be  6  (i.e., 
1x6).    Thus  the  ratio  could  be  expressed  as  the  achieved  score  divided 
by  possible  score  of  6/12  =  0.50.    Ratios  were  calculated  for  each  total 
component  and  for  planning  and  operational  procedures  within  each  component. 

Scores  for  a  site  can  be  interpreted  as  follows: 

Site  Total  Score  -  indicates  the  degree  to  which  the  site  has 
achieved,  documented,  systematized,  and  operational!' zed  ACQAP 
model  procedures  across  all  components;  i.e.,  the  comprehensive- 
ness of  the  quality  assurance  review  system. 

Site  Total  Planning  Score  -  indicates  the  degree  to  which  the 
site  has  documented  systematic  planning  procedures  across  all 
components  outlined  in  the  ACQAP  Model. 

Site  Total  Operational  Score  -  indicates  the  degree  to  which 
the  site  has  systematically  implemented  ACQAP  Model  operational 
procedures  across  all  components;  that  is,  the  degree  to  which 
operation  schedules  and  time  frames  are  documented,  predictable, 
and  made  explicit. 

Site  Component  Score  -  indicates  the  degree  to  which  a  site  has 
achieved  the  ACQAP  procedures  for  the  particular  area  of  activity 
(i.e.,  component);  indicates  relatively  strong  and  weak  areas  of 
the  total  program.    Component  scores  are  interpreted  as  total 
planning  and  operational  scores  above  but  for  one  component  only. 

Site  Procedure  Score  -  indicates  the  degree  to  which  a  site  has 
achieved  the  particular  ACQAP  procedure  (requirement);  indicates 
relatively  strong  and  weak  procedures  of  the  component  or  total 
program.    Procedure  scores  are  interpreted  as  total  planning  and 
operational  score  as  above  but  for  one  procedure  only. 


LIMITATIONS  AND  FUTURE  REFINEMENT 

The  ACQAP  Model  remains  in  a  developmental  stage.    The  validity  of  the 
model  must  continue  to  be  examined  in  terms  of  its  ability  to  distinguish 
between  programs  which  effectively  improve  medical  care  and  health.  Several 
types  of  validity  are  important  and  are  discussed  here:    content  validity, 
a  pragmatic  approach  to  concurrent  and  predictive  validity,  and  construct 
validity. 

At  this  point,  initial  content  validity  (e.g.,  a  systematic  comparison 
of  the  model  goals  and  procedures  in  ambulatory  care  quality  assurance 
programs)  appears  acceptable.    Content  was  reviewed  independently  by  fifteen 
consultants  and  comments  were  incorporated  into  the  final  model.    Also,  the 
model  has  been  applied  to  twenty-four  quality  assurance  review  programs 
without  encountering  major  content  problems. 
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However,  refinement  must  continue  to  focus  on  revision  of  goals, 
component  names,  and  procedures.    There  appears  to  be  a  need  to  examine 
word  usage,  specificity,  and  completeness.    For  example,  a  planned  change 
such  as  the  one  below  increases  specificity  and  measurabil ity: 

Old  procedure:    To  select  topics  or  providers  for  review 

New  procedure:    To  assure  topics  are  selected  on  the  basis  of: 

a.  Amenability  of  medical  information 

b.  Prevalence 

c.  Risk  to  patients 

d.  Potential  achievement  of  cost  and  health  benefit 

e.  Potential  for  change  through  feedback  procedures 

In  terms  of  completeness,  investigators  involved  in  the  survey  informally 
noted  potential  procedure  areas  which  are  either  not  currently  contained  in 
the  model  or  cannot  be  distinguished  in  the  model.    These  areas,  listed  below, 
will  be  examined  during  the  refinement  phase. 

1.  Most  programs  in  planning  quality  review  activities  anticipated  a 
consistent  activity  level,  usually  in  terms  of  number  of  topics  to  be 
reviewed,  amount  of  data  to  be  collected  and  physician  time  to  be  allocated 
to  the  program.    Once  programs  became  operational,  however,  fluctuations 

in  the  amount  of  quality  assurance  activity  usually  occurred  over  long  periods 
(six  to  twenty-four  months).    This  effect  appears  to  be  directly  related 
to  the  service  demands  placed  on  the  resources  of  the  medical  practice. 
Unanticipated  fluctuation  in  quality  assurance  activity  was  also  related 
to  the  low  priority  of  quality  assurance  programs  in  many  of  the  practices. 
Quality  review  was  often  viewed  as  a  function  that  could  be  reduced  or 
completely  stopped  for  a  period  of  time  without  having  any  effect  on  the 
quality  of  services  delivered.    The  current  model  does  not  measure  this  type 
of  activity  or  fluctuations  which  may  occur. 

2.  Goals  among  programs  varied  considerably.    Cost  and  utilization  control 
was  the  primary  goal  in  some  programs,  others  established  improved  quality 
of  medical  care  as  the  most  important  goal.    Some  programs  were  conceived 

as  research  projects  with  methodologic  development  and  improvement  as  the 
primary  focus.    Others  were  interested  in  establishing  an  ongoing  program 
as  part  of  the  operation  of  their  group  practice.    The  current  model  fails 
to  distinguish  among  types  of  goals  and  there  is  no  mechanism  for  measuring 
the  influence  of  goal  conflict  on  the  success  of  the  program. 

3.  Realistic  budgeting,  development  of  appropriate  organizational  structure 
and  design  of  effective  management  plans  within  the  organization  were 
important  to  quality  assurance  program  success.    The  model  is  not  equipped 

to  provide  the  data  needed  to  make  qualitative  assessment  of  administrative 
functions. 
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4.  Professional  and  administrative  staff  involvement  and  commitment 
in  the  planning  process  and  operational  phases  appear  to  be  essential 
for  program  success.    The  model  does  not  facilitate  assessment  of  staff 
involvement,  especially  in  the  planning  phase  where  documented  plans  are 
the  only  source  of  information. 

5.  Group  practices  with  the  most  success  in  developing  quality  assurance 
programs  were  those  that  had  reached  fiscal  and  organizational  stability 
before  the  review  program  was  implemented.    The  model  currently  does  not 
measure  stability  of  the  delivery  setting. 

6.  Successful  quality  assurance  program  operations  usually  employed 

a  key  person  to  direct  the  program  and  thereby  increased  staff  commitment 
to  the  program  as  well  as  consistency  and  continuity  of  program  activity. 

7.  Many  of  the  quality  assurance  programs  appear  to  be  viewed  as  projects 
rather  than  ongoing  functions  of  the  group  practice.    This  suggests  that 
ambulatory  care  quality  assurance  within  many  of  the  practices  is  at  the 
initial  stage  of  evolving  to  a  routine  function. 

Another  area  of  refinement  will  be  the  attempt  to  incorporate  an 
additional  activity  into  the  model.    This  activity  is  "topic  verification." 
Verification  of  the  topic  chosen  for  review  may  be  seen  as  a  pilot  or 
feasibility  study.    It  occurs  after  a  quality  assurance  program  has  identified 
a  potential  topic  for  review,  but  before  it  commits  the  sizable  resources 
that  are  required  by  the  review  process. 

Sample  data  are  collected  and  analyzed  to  verify  that  the  potential 
topic  chosen  for  review  is  likely  to  be  a  problem  upon  extensive  review 
(i.e.,  a  more  complete  set  of  criteria  and  more  cases).    Screening  criteria 
are  developed  (or  taken  from  a  larger  list  developed  for  that  topic), 
based  on  their  ability  to  provide  indications  of  problem  areas.  Data 
elements  for  the  measurement  of  patient,  provider  or  institutional  behavior 
are  identified  and  data  are  collected  for  a  set  of  cases.    A  reviewer 
decides  whether  a  problem  exists  for  each  case  by  comparing  its  data  profile 
to  the  criteria.    The  topic  verification  judgment  is  based  upon  inferring 
from  the  pilot  data  an  expectation  of  the  prevalence  and  seriousness  of 
the  problem  in  the  community.    A  decision  is  made  concerning  the  utility 
of  committing  the  resources  required  for  a  full  audit. 

Some  degree  of  pragmatic  validity  for  determining  the  usefulness 
of  the  measuring  instrument  as  an  indicator  (concurrent  validity)  or  predictor 
(predictive  validity)  exists  in  the  model  since  quality  assurance  procedures 
and  programs  which  received  high  scores  on  the  model  were  those  which, 
in  the  opinion  of  investigators  not  involved  in  scoring,  were  most  systematic, 
comprehensive  and  acceptable  in  terms  of  procedures  measured. 
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Further  pragmatic  validation  will  be  examined  through  two  techniques. 
First,  correlation  of  scores  assigned  independently  by  experts  assumed 
to  be  able  to  judge  the  validity  of  a  program  in  terms  of  its  data,  criteria 
and  medical  decision-making  with  scores  assigned  by  "nonexperts"  add  insight 
to  the  usefulness  of  the  tool  for  measuring  existing  program  attributes. 
The  second  technique  is  to  correlate  program  scores  to  the  degree  of  impact 
the  program  has  on  what  it  is  designed  to  affect  (e.g.,  physician  behavior, 
consumer  behavior,  quality  of  medical  care,  etc.).    This  method  requires 
careful  definition  of  the  goals  of  quality  assurance  review  and  the  construct 
of  "quality."    It  leads  the  validation  process  into  the  area  of  construct 
validity  (i.e.,  the  degree  to  which  scores  obtained  upon  use  of  the  model 
form  a  pattern  consistent  with  what  is  known  about  the  construct  of  "quality" 
and  its  assurance).    Construct  validity  is  the  most  difficult  to  establish 
because  of  the  vague  and  varied  definitions  of  quality  of  medical  care 
as  well  as  the  different  purposes  for  implementing  quality  assurance  programs. 

Plans  for  model  refinements  in  the  next  stage  of  development  focus 
on  the  validity  questions  described.    The  model  will  be  applied  and  tested 
in  six  ambulatory  review  systems.    The  goal  is  to  refine  accuracy  of  model 
content  and  scoring  procedures  to  further  define  what  scores  on  the  model 
indicate  or  predict,  and  to  develop  methods  for  use  of  the  model  as  an 
assessment  tool.    Empirical  methods,  rather  than  normative,  must  be  developed 
to  test  and  define  the  minimum  set  of  procedures  to  be  included  in  the 
model . 


POTENTIAL  APPLICATIONS 

Given  the  discussion  of  limitations,  it  appears  that  the  ACQAP  Model 
could  be  used  for  three  types  of  ambulatory  quality  assurance  activity: 
(1)  design  and  development  of  quality  review  systems,  (2)  assessment  of 
operational  review  systems  and  (3)  descriptive  studies. 

The  ACQAP  Model,  with  its  list  of  procedures  which  must  be  achieved, 
is  a  useful  tool  for  guiding  the  design  and  development  of  ambulatory 
care  quality  assurance  systems.    If  used  in  a  subjective  checklist  fashion, 
the  model  helps  to  assure  that  specific  procedures  are  not  overlooked, 
that  a  systematic,  sequential  approach  is  used  in  the  development  phase, 
and  that  communication  among  system  developers  is  structured. 

An  example  of  the  manner  in  which  the  model  can  be  used  as  a  check- 
list is  presented  in  Figure  6,  "System  Development  Checklist."  Model 
Component  9,  as  it  is  developed,  is  subjected  to  review  in  order  to  deter- 
mine whether  model  procedures  are  being  planned,  developed  or  implemented. 

As  progress  is  made,  each  procedure  is  checked  to  indicate  progress 
made  for  that  procedure.    If  progress  towards  implementation  appears  inadequate, 
attention  to  the  procedure  and  the  causes  for  its  delay  can  be  analyzed. 
The  model,  when  used  frequently  in  this  manner,  will  help  to  assure  thorough 
planning  and  implementation  of  quality  assurance  procedures. 
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The  ACQAP  Model  also  appears  to  have  potential  for  use  as  an  assess- 
ment tool.    In  its  present  form,  assessment  of  system  activity  (e.g., 
the  frequency  of  procedure  implementation)  is  possible  through  use  of 
a  checklist  format  such  as  that  described  in  Figure  6.    The  next  step 
in  the  assessment  process  must  focus  not  only  on  whether  the  procedure 
was  accomplished  but  on  how  well  it  was  achieved.    The  present  model  does 
not  contain  indicators  of  how  well  procedures  are  achieved;  the  quality 
assurance  system  itself  must  set  standards  applicable  to  its  unique  program 
and  assess  itself  against  these  standards. 

Not  every  ACQAP  Model  procedure  lends  itself  equally  well  to  assess- 
ment in  the  form  of  standard  setting.    For  example,  it  is  more  feasible 
to  set  a  standard  for  a  procedure  which  states,  "To  assure  that  the  data 
collection  and  processing  procedures  are  reliable,"  than  for  a  procedure 
such  as,  "To  determine  the  validity  of  data  sources."    Therefore,  standards 
probably  would  not  be  set  for  all  procedures  or  the  degree  of  specificity 
would  vary  according  to  the  goals  of  the  quality  assurance  system.  One 
major  objective  of  the  refinement  phase  will  be  to  develop  procedures 
along  with  standards  which  are  uniformly  applicable  across  ambulatory 
quality  assurance  programs. 

Finally,  the  ACQAP  Model  could  be  used  as  a  tool  for  conducting  a 
descriptive  study  to  analyze  factors  affecting  quality  assurance  systems. 
For  example,  one  may  wish  to  know  how  the  size  of  a  physician  staff  affects 
the  implementation  of  a  quality  assurance  review  program.    Assuming  the 
ACQAP  Model  is  applied  and  scored  as  in  the  survey,  specific  analysis 
to  identify  factors  contributing  to  high  or  low  scores  is  possible.  The 
next  section  contains  a  case  study  illustrating  one  possible  descriptive 
study  method.    The  results  of  such  a  case  study  help  to  indicate  factors 
which  might  be  considered  when  implementing  a  program  with  a  large  or 
small  staff. 

In  conclusion,  application  of  the  ACQAP  Model  has  potential  in  three 
areas.    First,  a  guide  to  system  design  and  implementation;  second,  as 
a  tool  for  program  assessment  and  third,  to  assist  in  studies  aimed  at 
analyzing  factors  within  programs.    It  is  recognized  that  the  model  requires 
further  refinement,  testing,  and  validation;  as  these  activities  continue, 
the  value  of  the  ACQAP  Model  for  system  design,  assessment,  and  study 
will  continue  to  increase. 


SAMPLE  CASE  STUDY:    PHYSICIAN  STAFF  SIZE  AND  AMBULATORY  QUALITY  ASSURANCE 
PROGRAMS. 

Question 

How  does  the  size  of  physician  staff  affect  the  development  of  quality 
assurance  program  plans  and  the  subsequent  implementation  of  those  plans? 
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Method 


The  purpose  of  this  case  study  is  to  illustrate  potential  applica- 
tions of  the  ACQAP  Model  to  the  analysis  of  quality  assurance  programs. 
Since  the  purpose  is  illustrative,  the  case  study  is  based  on  the  analysis 
of  four  programs  to  simplify  the  presentation.    A  larger  sample  would  be 
required  to  reach  more  definitive  conclusions  if  this  case  study  were  to 
be  replicated.    Physician  staffing  is  defined  as  the  number  of  physicians 
employed  in  a  setting  to  deliver  health  care.    For  this  definition,  no 
distinction  is  made  between  part-time  and  full-time  physicians  since  it 
is  assumed  that  for  quality  assurance  activities  to  be  comprehensive  those 
activities  must  address  all  physicians  within  a  setting.    Four  quality 
assurance  program  sites  are  analyzed:    large  staff  sites  A  and  B  had  65 
and  78  physician  staff  members  and  small  staff  sites  Y  and  Z  had  4  and  6 
physician  staff  members.    All  sites  have  been  in  operation  for  the  same 
amount  of  time. 

Components  III  (Selection  of  Topics  and  Providers  for  Review)  and 
VIII  (Decision  Making  and  Feedback)  of  the  ACQAP  Model  were  chosen  since 
the  general  goals  of  these  two  components  and  specific  procedures  relate 
directly  to  physician  involvement  in  quality  assurance  activities.  Within 
each  component,  procedures  were  chosen  to  more  specifically  analyze  quality 
assurance  activities  in  which  physicians  or  total  physician  staff  size 
seemed  critical  to  the  planning  and  operation  of  a  quality  assurance 
program. 

The  analysis  examined  two  areas: 

a.  The  difference  for  planning  scores  between  sites  with  large 
and  small  physician  staffs 

b.  The  difference  for  operational  scores  between  sites  with 
large  and  small  physician  staffs 

The  purpose  of  analysis  is  to  discuss  and  note  whether  there  are 
score  differences,  as  well  as  other  key  differences  based  on  knowledge 
of  the  four  sites. 


Analysis  and  Discussion 

An  examination  of  scores  for  the  four  sites  reveals  differences  for 
both  Component  III  and  Component  VIII.    Illustrated  in  Table  I,  Component 
III,  which  requires  sites  to  identify  topics  for  quality  assurance  review, 
received  higher  total  scores  at  the  sites  with  large  physician  staffs 
(A  and  B)  than  did  the  two  sites  with  small  physician  staffs  (Y  and  Z). 

This  result  suggests  that  sites  with  smaller  physician  staffs  had  more 
difficulty  accomplishing  Component  III.    The  difference  between  the  two 
categories  becomes  more  perceptible  when  one  examines  the  difference  in 
Component  III  operational  scores.    The  score  difference  could  be  attributed 
to  the  fact  that  investigators  documented  the  lack  of  administrative  approach 
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to  implementing  quality  assurance  activities  in  the  small  staff  sites.  On 
the  other  hand,  the  two  larger-staffed  sites,  according  to  site  investigators, 
emphasized  a  more  systematic  administrative  approach  in  dealing  with  staff 
physicians  and  their  participation  in  a  program.    This  view  is  substantiated 
when  briefly  comparing  the  total  scores  of  each  staff  size  group  for 
Component  II  (Establishment  of  Administrative  Procedures)  in  Table  II. 


TABLE  I 


COMPONENT  III:    SELECTION  OF  TOPICS  AND  PROVIDERS  FOR  REVIEW 
PLANNING  AND  OPERATION  SCORES 


Site 

Planning 

Operation 

Total 

A 

1.00 

1.00 

1.00 

D 
D 

l  no 

U .  oo 

u. 

Av.  Score  - 

High  1.00 

0.94 

0.97 

Y 

0.75 

0.50 

0.63 

Z 

1.00 

0.13 

0.57 

Av.  Score  - 

Low  0.88 

0.32 

0.60 

TABLE 

II 

COMPONENT  II: 

ESTABLISHMENT  OF 

ADMINISTRATIVE 

PROCEDURES 

PLANNING  AND  OPERATION  SCORES 

Site 

Planning 

Operation 

Total 

A 

0.65 

0.75 

0.70 

B 

0.50 

0.50 

0.50 

Av.  Score  - 

High  0.58 

0.62 

0.60 

Y 

0.25 

0.40 

0.33 

Z 

0.05 

0.10 

0.08 

Av.  Score  - 

Low  0.15 

0.25 

0.20 
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Component  II  measures  a  site's  establishment  of  an  administrative 
structure  to  develop  and  implement  procedures  for  operating  quality 
assurance  activities.    The  component  requires  sites  to  plan  and  implement 
20  procedures.    Site  scores  are  shown  in  Table  II.    The  two  larger  staffed 
sites'  total  component  scores  were  0.70  and  0.50,  while  the  two  smaller 
staffed  sites'  total  scores  were  0.33  and  0.08.    This  disparity  in  scores 
further  suggests  and  supports  the  investigators'  perception  of  differences 
in  the  administrative  approach. 

The  difference  between  the  two  groups'  quality  assurance  programs, 
as  related  to  physician  staff  size,  becomes  more  evident  when  analyzing 
Component  VIII.  This  component  requires  sites  to  plan  and  implement 
quality  assurance  procedures  which  could  acutely  affect  physicians'  activity 
within  a  setting.  The  overall  goal  of  Component  VIII  is  to  develop  and 
implement  feedback  procedures  based  on  decisions  made  during  the  review 
process . 

The  distribution  of  scores  for  the  four  sample  sites  is  as  follows: 


TABLE  III 


COMPONENT  VIII 

:  DECISION 

MAKING  AND  FEEDBACK 

PLANNING 

AND  OPERATION  SCORES 

Site 

Planning 

Operation 

Total 

A 

0.56 

0.67 

0.62 

B 

0.63 

0.44 

0.54 

Av.  Score  -  High 

0.60 

0.56 

0.58 

Y 

0.37 

0.17 

0.27 

Z 

0.13 

0.00 

0.07 

Av.  Score  -  Low 

0.25 

0.08 

0.17 

An  examination  of  the  average  planning  scores  reveals  differences  between 
the  groups.    The  larger  staffed  sites  were  more  successful  in  planning 
feedback  procedures  to  physicians  than  the  two  smaller  staffed  sites. 
Component  VIII  operational  and  total  scores  draw  an  even  sharper  distinc- 
tion for  the  sites.    The  sites  with  large  physician  staffs  were  far  more 
successful  in  implementing  their  plans  for  feedback,  while  the  two  smaller 
staffed  settings  had  negligible  feedback  activity.    Site  investigators 
noted  that  the  larger  staffed  sites  had  well-established  physician  popu- 
lations with  low  turnover  rates.    The  two  smaller  staffed  sites  were  in 
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the  midst  of  physician  changes  and  recruitment.    The  difference  in  both 
planning  and  operation  scores  for  Component  VIII  suggests  that  stability 
of  physician  staff  may  be  a  key  variable  when  quality  assurance  programs 
advance  to  the  point  where  systematic  feedback  of  review  results  are  important 
to  the  overall  success  of  a  program.    Further,  to  score  high  on  Component 
VIII,  as  the  two  larger  sites  did,  requires  a  program  to  consciously  plan 
and  cultivate  physician  support.    This  variable  of  support,  reflected  in 
scores,  was  documented  by  investigators.    The  documentation  indicated  that 
the  two  larger  staff  sites  emphasized  the  cultivation  of  physician  support 
as  an  important  element  for  the  entire  program.    Further,  to  organize  support 
from  the  larger  staffs  required  these  programs  to  commit  substantial 
resources  to  that  activity  as  a  part  of  the  quality  assurance  program. 
The  smaller  staffed  sites  indicated  staff  support  was  also  critical,  although 
investigators  could  not  document  efforts  in  this  area.    It  is  possible 
that  quality  assurance  programs  for  large  physician  staffs  require  more 
systematic  planning  and  a  more  formal  approach.    It  may  be  that  a  program 
with  a  small  physician  staff  will  use  less  formal  administrative  activities 
for  implementation,  as  well  as  plan  less.    The  scores  for  these  four  sites 
support  such  a  view. 

The  question  of  physician  staff  size  can  be  analyzed  further  by  looking 
at  the  difference  in  scores  for  specific  Component  III  and  VIII  procedures. 
For  Component  III,  three  procedures  dealing  specifically  with  physicians ' 
role  in  quality  assurance  activities  were  chosen,  one  planning  (B)  and  two 
operational  (D  &  E): 

Procedure  B:    To  "develop  guidelines  for  the  selection  of 

providers  for  topic  review. 
Procedure  D:    To  select  providers  for  review. 
Procedure  E:    To  assure  cases  reviewed  are  representative 

of  the  total  number  of  topics  and  providers 

under  review. 

Table  IV  presents  site  scores  on  each  procedure.    There  does  not  seem  to 
be  a  substantial  difference  for  Procedures  B  and  D.    However,  in  analyzing 
the  Procedure  E  scores  for  the  four  sites,  we  see  a  significant  difference 
between  the  larger  and  smaller  staffed  sites.    Because  this  procedure 
requires  sites  "to  assure  cases  reviewed  are  representative  of  a  total 
number"  under  review,  one  would  expect  differences  between  the  two  groups. 
The  scores  indicate  that  the  two  large  staffed  sites  were  successful  and 
the  two  small  staffed  sites  unsuccessful  in  implementing  Procedure  E. 
It  is  doubtful  whether  these  scores  indicate  that  it  is  easier  to  accomplish 
this  procedure  with  larger  staffs;  however,  investigators  note  that  the 
Site  A  and  B  programs  were  much  more  sensitive  and  organized  in  approaching 
the  issues  contained  in  Procedure  E.    Sites  Y  and  Z,  though  having  much 
smaller  staffs,  did  not  consider  the  questions  involved  in  Procedure  E. 
The  scores  seem  to  indicate  that  smaller  staff  sites  do  not  formalize 
programs  to  the  extent  that  larger  staffed  sites  do.    Again,  these  scores 
substantiate  the  points  made  earlier  regarding  the  difference  in  formalizing 
elements  of  a  quality  assurance  program  for  the  two  groups. 
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TABLE  IV 

COMPONENT  III:    SELECTION  OF  TOPICS  AND  PROVIDERS  FOR  REVIEW 

PROCEDURE  SCORES 


r j oceuure  ocores 

B 

D  E 

A 

2 

2  2 

B 

2 

2  2 

Y 

1 

2  0 

Z 

2 

1  0 

Component  VIII  planning  and  operational  procedures  chosen  for  analysis 
include  the  following: 

Procedure  A:    To  develop  guidelines  for  timely  and  acceptable 

feedback  procedures. 
Procedure  G:    To  develop  guidelines  to  allow  providers  to 

appeal  (review)  decisions. 
Procedure  I:    To  implement  timely  and  acceptable  feedback 

procedures. 

Procedure  P:    To  allow  providers  to  appeal  presented  (review) 
decisions. 

Scores  for  procedures  A,  G,  I,  and  P  are  displayed  in  Table  V. 

TABLE  V 

COMPONENT  VIII:    DECISION  MAKING  AND  FEEDBACK 
PROCEDURE  SCORES 


Site 

Procedure 

Scores 

A 

G 

I 

P 

A 

1 

0 

1 

0 

B 

2 

2 

1 

1 

Y 

2 

0 

0 

0 

Z 

0 

0 

0 

0 
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Although  some  differences  between  groups  exist  in  planning  Procedures 
A  and  G,  only  the  large  staffed  sites  were  relatively  successful  in  accom- 
plishing the  operational  Procedure  I  and  P.    Only  one  small  staffed  site 
received  any  score  besides  zero  across  the  four  procedures.    Again,  there 
appears  to  be  a  lack  of  systematic  implementation  of  physician-oriented 
procedures  in  the  sites  with  a  small  staff  size. 


Summary 

This  analysis  attempted  to  explore  the  issue  of  physician  staff  size 
and  its  effect  on  the  development  and  implementation  of  two  quality 
assurance  components  and  the  associated  specific  procedures.    The  components 
and  procedures  selected  for  analysis  were  perceived  to  be  highly  related 
to  physician  participation  in  quality  assurance  programs,  and  thus  related 
to  the  question  of  whether  staff  size  affects  systematic  operational  status 
of  the  program. 

The  preceding  analysis,  predominantly  based  on  ACQAP  Model  scores, 
indicates  that  when  larger  physician  populations  are  involved  in  a  review 
process,  program  components  are  planned  and  operationalized  in  a  more 
formal  manner  than  at  smaller  staffed  sites.    The  larger  staffed  settings 
with  more  complicated  administrative,  communication  and  personnel  interaction 
patterns  may  score  higher  because  their  complex  situation  requires  more 
formal  planning  and  documentation  of  the  quality  assurance  system.  It 
may  be  that  quality  assurance  programs  in  smaller  staffed  settings  are  more 
informal  and  less  routine  due  to  the  close  working  relationship  providers 
have  with  their  peers. 

The  implications  of  this  limited  analysis  are:    (1)  If  smaller  settings 
are  required  to  formalize  their  quality  assurance  activity  for  regulatory 
purposes,  there  may  be  some  inclination  by  those  settings  to  view  such 
requirements  as  unnecessary.    Effort  may  be  required  to  motivate  personnel 
to  approach  quality  assurance  activity  in  a  more  formal  manner.    (2)  Larger 
settings  may  require  formalized,  documented  systems  to  assure  that  quality 
assurance  activity  will  routinely  measure  physician  performance.  Formalized 
effort  may  be  required  to  implement  communication  systems  designed  to 
increase  understanding  of  the  purpose,  methods,  and  results  of  the  program. 
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ACQAP  MODEL  SCORES  ACROSS  SURVEY  SITES 


This  section  presents  and  discusses  scores  that  were  assigned  to 
24  survey  sites.    Scores  are  based  on  procedures  outlined  in  the  ACQAP 
Model.    Although  a  total  of  27  sites  was  included  in  the  project,  3  sites 
did  not  receive  scores  since  they  were  classified  as  research  projects 
whose  activity  was  aimed  at  only  a  limited  number  of  procedures  contained 
in  the  ACQAP  Model. 

The  purpose  of  examining  scores  is  to  assist  in  defining  hypotheses 
about  ambulatory  care  quality  assurance  programs.    These  hypotheses  are 
expected  to  be  relevant  to  future  model  refinement  and  program  testing  by 
federal  and  private  groups.    The  findings  of  the  report  are  based  only  on 
the  24-site  sample,  which  is  biased  by  the  fact  that  sites  were  selected 
for  their  operational  status  and  their  agreement  to  participate.  Findings 
are  not  intended  for  further  generalization  but  rather  for  further  investi- 
gation. 

Table  VI,  "Summary  of  Scores  by  Site,"  presents  the  scores  for  each  of 
the  24  sites.    Individual  Planning  and  Operation  scores  ranged  from  0.00, 
the  lowest  possible  score,  to  1.00,  the  highest  possible  score.  Across 
all  sites,  the  average  Planning  score  was  0.60;  mean  Operation  score  was 
0.52  and  the  mean  Total  score  was  0.56.    Table  VI  forms  the  basic  set  of 
scores  discussed  in  this  section. 


COMPONENT  SCORES  ACROSS  SITES 

The  average  Total,  Planning  and  Operation  scores  for  each  component 
across  all  sites  offer  a  means  of  ranking  the  nine  ACQAP  Model  components. 
This  type  of  analysis  is  useful  in  determining  which  components  might 
require  special  assistance  during  implementation  of  quality  assurance 
programs . 

Figure  7,  "Ranked  Average  Component  Score  Across  All  Sites,"  repre- 
sents graphically  the  ranks  of  components.    In  terms  of  component  Planning 
scores,  Component  IX  (Evaluation,  Restudy,  and  Further  Action)  and  VIII 
(Decision  Making  and  Feedback)  rank  lowest  with  Component  VII  (Review 
Procedures)  and  Component  II  (Establishment  of  Administrative  Procedures) 
also  scoring  low.    Operation  scores  indicate  that  Component  IX  and  VIII 
again  are  lowest.    However,  Component  V  (Identification  of  Data  Elements 
and  Sources)  and  Component  IV  (Criteria  Development)  also  scored  low. 
Across  all  sites  the  components  that  scored  highest  for  Planning  Score  were 
Component  V  (Identification  of  Data  Elements  and  Scores)  and  Component 
III  (Selection  of  Topics  and  Providers  for  Review).    The  components  that 
scored  highest  on  Operation  Score  were  again  III,  but  also  VI  (Data  Collection 
and  Processing). 

The  relationship  between  Planning  and  Operation  scores  is  illustrated 
in  Figure  8,  "Average  Scores  for  Each  Component  Across  All  Sites."  Since 
Component  I  has  no  planning  procedures,  no  relationship  between  Planning 
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RANKED  AVERAGE  COMPONENT  SCORE  ACROSS  ALL  SITES 
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and  Operation  scores  is  shown.    It  is  noted  that  planning  procedures  scored 
higher  than  operation  procedures  for  Components  III  thru  VI;  operation 
procedures  scored  higher  for  Components  VII  thru  IX. 

Generally,  there  appears  to  be  a  close  relationship  between  Planning 
and  Operation  scores  except  for  Component  V  (Identification  of  Data  Elements 
and  Sources).    Component  V  shows  the  largest  Planning  and  Operation  score 
discrepancy  of  all  components.    Across  all  sites  the  average  component  V 
Planning  score  is  0.95  while  the  average  Operation  score  is  only  0.45. 

The  reasons  for  the  disparity  in  Component  V  may  be  related  to  the 
ease  with  which  the  planning  procedures  are  achieved,  thus  resulting  in 
a  high  Planning  score,  while  operation  activities  are  not  as  easily  achieved; 
hence  a  low  score  results.    Component  V  planning  procedures  are: 

a.  To  develop  guidelines  for  the  selection  of  data  elements  and 

b.  To  develop  guidelines  for  determining  the  data  sources  from 
which  data  elements  can  be  selected. 

Determining  data  elements  and  sources  is  a  relatively  easy  process  and  is 
an  obvious  necessity  in  conducting  a  quality  assurance  program,  thus  it 
generally  achieves  a  high  score. 

However,  implementation  of  these  objectives  appears  to  be  more  diffi- 
cult.   The  Component  V  operational  procedures  are: 

c.  To  select  the  data  elements  from  appropriate  sources 

d.  To  determine  the  reliability  of  data  sources 

e.  To  determine  the  validity  of  the  data  sources 

According  to  the  Procedures  scores  discussed  in  the  next  section  Procedure  c) 
is  achieved  more  often  than  either  d)  or  e);  therefore,  d)  and  e)  tend  to 
bring  down  the  Operational  score  for  the  component,  resulting  in  the  low 
Operational  score  and  reflected  finally  in  the  low  relationship  between 
planning  and  operation. 

In  conclusion,  it  appears  from  average  component  scores  that  the 
ambulatory  care  quality  assurance  programs  surveyed  planned  most  in  areas 
focusing  on  "Identification  of  Data  Elements  and  Sources"  and  "Selection  of 
Topics  and  Providers  for  Review."    "Selection  of  Topics  and  Providers  for 
Review"  was  then  implemented  most  along  with  "Data  Collection  and  Processing." 
However,  although  data  are  being  collected  and  processed,  it  appears  that 
lack  of  checks  of  data  reliability  and  validity  caused  a  low  Operation 
score  for  "Identification  of  Data  Elements  and  Sources."  "Criteria  Develop- 
ment" also  scored  low.    The  lowest  Planning  and  Operation  scores  were  on 
"Decision  Making  and  Feedback"  and  "Evaluation,  Restudy,  and  Further  Action." 

Data  collection  and  processing  consume  a  major  part  of  the  technical 
resources  in  many  quality  assurance  programs.    Low  operation  scores  on 
procedures  to  obtain  valid  and  reliable  information  as  well  as  criteria 
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development  and  relatively  high  planning  scores  on  data  collection  and 
processing  procedures  suggest  that  emphasis  in  these  two  areas  should 
be  reversed  in  many  of  the  sites  included  in  the  survey.    If  the  scoring 
for  this  sample  were  to  hold  for  a  larger  representative  sample  of  ambulatory 
quality  assurance  programs  it  would  suggest  that  developmental  priorities 
should  be  shifted  from  data  collection  and  processing  to  improvement  of 
criteria  development  and  the  basic  medical  records  and  the  data  sources 
used  for  review  and  assessment  of  the  quality  of  ambulatory  medical  care. 


PROCEDURE  SCORES  ACROSS  SITES 

To  assist  the  reader  in  acquiring  a  more  detailed  understanding  of 
the  scores  received  on  procedures  within  components,  a  ranking  scheme 
similar  to  the  one  outlined  in  the  preceding  section  is  presented.  Two 
sites  were  excluded  from  analysis  because  of  late  arrival  of  data.  As 
the  highest  possible  score  on  a  procedure  is  two  points,  across  the  22 
sites  the  highest  possible  total  score  for  each  procedure  is  44  points 
(e.g.,  22  x  2).    When  considering  points  scored  on  planning  and  operation 
procedures  combined,  Figure  9  illustrates  that  very  few  procedures  (seven) 
scored  the  total  possible  44  points.    Overall,  on  the  99  procedures  contained 
in  the  model,  only  about  half  the  total  possible  points  were  scored. 


FIGURE  9 

NUMBER  OF  POINTS  SCORED  ON  THE  99  PROCEDURES  IN  THE  ACQAP  MODEL 
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Figure  10  presents  the  ranked  planning  procedures.    The  ACQAP  Model 
contains  a  total  of  39  planning  procedures.    Only  11  procedures  received 
at  least  33  of  the  44  total  possible  points.    Twenty-three  planning  procedures 
received  22  or  less  of  the  44  total  possible  points;  these  low  scoring 
planning  procedures  are  listed  in  Figure  11.    The  ACQAP  Model  contains 
a  total  of  60  operation  procedures;  they  are  ranked  in  Figure  12.  Only 
14  procedures  received  at  least  33  of  the  44  total  possible  points.  Twenty- 
seven  procedures  received  22  or  less  of  the  44  total  possible  points; 
those  low  scoring  operation  procedures  are  listed  in  Figure  11.    It  is 
hypothesized  that  the  low  scoring  procedures  listed  in  Figure  11  should 
receive  more  attention  during  program  planning  and  operation. 

As  discussed  previously,  many  of  the  low  scoring  procedures  focus 
on  determining  reliability,  accuracy  and  consistency  of  data  and  review 
decisions  as  well  as  the  degree  to  which  the  data  and  review  decisions 
are  valid  (e.g.,  based  on  data  that  accurately  reflect  physician-patient 
encounters).    Feedback  and  restudy  also  contained  many  low-scoring  pro- 
cedures.   These  two  components  measure  intervention  activity  and  resulting 
improvement.    The  model  itself  is  limited  since  it  does  not  directly  measure 
cause  and  effect,  however,  from  the  low  scoring  procedures  it  might  be 
hypothesized  that  one  reason  for  lack  of  feedback  and  restudy  might  be 
lack  of  confidence  in  the  data  presented  for  review  and  therefore,  a  reluc- 
tance to  contact  physicians  and  feed  back  the  results  of  a  medical  review. 
More  attention  to  determining  data  accuracy  may  increase  likelihood  of 
frequent  feedback  and  restudy  designed  to  measure  impact  on  change  in 
physician  behavior. 

Low  scores  for  the  decision  making  and  feedback  component  might  also 
be  due  to  inherent  reluctance  to  confront  peer  groups  or  individuals  with 
evidence  of  substandard  care  or  because  little  is  known  about  effective 
methods  to  modify  group  and  individual  behavior.    Because  of  the  relatively 
consistent  low  scores  across  all  sites  for  the  decision  making  and  feedback 
component,  most  programs  could  be  characterized  as  review  programs  as 
opposed  to  quality  assurance  programs. 

If  low  scores  for  the  decision  making  and  intervention  component 
were  found  in  a  large  and  representative  sample  of  quality  assurance  programs 
this  characterization  might  well  typify  the  state  of  the  art  in  ambulatory 
care  quality  assurance  and  account  for  the  investigator  observation  that 
most  of  the  programs  in  this  survey  were  viewed  from  within  the  practice 
as  projects,  as  opposed  to  routine  and  essential  functions  of  the  group. 
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SCORES  BY  SETTING  AND  QUALITY  ASSURANCE  PROGRAM  FACTORS 


To  formulate  hypotheses  regarding  the  factors  within  ambulatory  quality 
assurance  settings  and  programs  that  may  influence  ability  to  score  higher 
on  the  ACQAP  Model  procedures,  15  categories  (within  6  factors)  were 
selected  for  investigation.    Figure  13  presents  the  factors  and  categories; 
definitions  were  previously  presented  in  Figure  3. 


FIGURE  13 

SETTING  AND  QUALITY  ASSURANCE  PROGRAM  FACTORS  AND  CATEGORIES 


Factor 


Categories 


1.    Type  of  Group  Practice 


2.    Type  of  Revenue  Source  to 
Delivery  System 


3.  Length  of  Quality 
Assurance  Program  Activity 

4.  Type  of  Quality 
Assurance  Program  Support 

5.  Type  of  Quality 
Assurance  Review 

6.  Type  of  Data  Source  for 
Quality  Assurance  Program 


Single  Specialty 
General  Practice 
Mul ti specialty 

Fee  for  Service 
Prepaid 

Both  FFS  and  PP 

0-24  Months 

25  Months  and  More 

Volunteer 
Budgeted 

Internal 
External 


Claim  Form 

Chronological  Medical  Record 
Problem  Oriented  Medical  Record 


It  was  decided  a  priori  to  group  each  site  into  either  a  high  range  (e.g., 
site  total  score  was  above  0.50)  or  a  low  range  (e.g.,  site  total  score 
was  0.50  or  below).    Table  VII  displays  the  percent  of  high  range  sites 
per  category  and  the  difference  in  percentage  points  between  categories. 

Quality  review  programs  which  involve  single  specialty  providers 
appear  to  score  high  when  compared  to  those  with  General  Practitioners  and 
Mul ti special ty  providers.    No  explicit  reasons  were  found  to  explain  the 
difference,  however,  it  could  be  hypothesized  that  topic  identification  and 
criteria  development  may  be  more  easily  achieved  due  to  a  more  focused 


37 


1 1 1 

t — i 

i  i  i 

1 1 1 

ZS  CO 

1 1  I 

CD 

1 —  UJ 

o 

<C 

1 1 1  1— 1 

r— 

CO  Od 

UJ 

o 

cc 

UJ 

CO  CD 

UJ 

c_> 

1—  UJ 

Ll_ 

ce: 

z  I— 

Ll_ 

UJ 

i — i 

Q_ 

o  c_> 

Q 

u_ 

UJ 

>- 

o 

CD 

C£  >- 

cc 

uj  oc: 

o 

1 — 

ct 

o_  0 

CD 

q; 

CD 

UJ 

UJ 

CO  UJ 

1— 

o 

3: 

UJ  t— 

<c 

q: 

CD 

K—  <=C 

o 

UJ 

1— i 

1— 1  0 

Q_ 

CO 

>- 

CO 

co 

UJ 

UJ 

 i 

1 — 

CO 

>- 

1 — 1 

1— 1 

1 — 1 

oc 

co 

CO 

0 

CO 

00 

CD 

UJ 

o 

UJ 

UJ 

CD 

1 — 

1— 

1 — 1 

<c 

<c 

CO 

0 

oc 

o 

•2C 

1 — 1 

o 

>> 

 1 

>- 

UJ 

Q 

~T~ 

00 

rv 

 1 

CD 

UJ 

O 

CO 

< 

1 — 1 

1— 

CD 

=E 

1— 1 

UJ 

1— 

UJ 

CO 

1— 

CD 

U_ 

<C 

O 

UJ 

O 

< 

CD 

oc: 

oc 

O 

UJ 

nr 

CD 

i— i 

IC 

Ll_ 

O 

>- 

o; 

o 

0 

CO 

CD 

1 — 1 

UJ 

oc 

h- 

<c 

Q_ 

<_> 

2: 

o 

o 

ct 
o 


o 


00  o 


<r>  p—  >vt- 


>>  cu 

4->  o 

■—•->> 
fO  +->  +-> 

•r-   O  r— 

O  (O  ro 
CU  S-  •■- 

0.0.  u 
CO  cu 
1—  a. 

CU  rC  to 

I—     S-  •!- 

CD  <U  -t-> 
C  E  r— 
•1—  CD  =3 
CO  CD  21 


O 

s- 

CD 

CU 

4-  O 
O  -r- 
4-> 

cu  a 
0.  ro 
,>>  s- 


CM 


<r>  1—  «^t- 


CM  CO  CM 


cu 

0 

•1 — 

> 

S- 

0) 

co 

S_  T3 

O  «r- 

4-  ro 

CL-CZ 

a>  <u 

+■» 

a)  s- 

0 

u_  o_ 

CO 

cu 

E 

a> 

e 

CO 


> 
a> 

CM  oc  o 

4-  S- 
O  <U  (U 

a  > 
cu  $-  1- 
a.  3 
>>  o  a> 

I —  CO  Q 


CM  r— 


VO  CT> 


CU 
S- 

o 


S- 
co  O 

•(->  co 
C  -C 
o  +-> 
2:  c 
o 
2: 

CNJ 

O  <M 


o  co 


CO  CM 


s- 

CU  T3 

a>  a> 

+■>  4-> 

c  cu 

3  CD 

■—  -a 
o  3 

>  CO 


*  5 

+■> 

s- 

<c  > 

0 

•  Q. 

O"  +-> 

<C  CL 

a 

•  3 

4-  < 

«nT  crco 

0 

E 

4—  £s 

-£=  ro 

0  <o 

4->  S- 

S. 

co  cn 

<D  cn 

c  0 

Q-  O 

cu  s- 

,>)  s- 

_J  Q_ 

1—  Q- 

CM  0O 


fO  «o 
c  c 

S-  S- 
0)  cu 
+->  +-> 

C  X 


LO  O" 


o  s 

cu 

CU  -i- 
Q.  > 

>,  cu 
H-  oc: 


CO  I —  CT> 


00  «d- 


S- 

o 


* 
* 

a: 


E  * 

•I-  * 
ro  a: 
1—  2:  o 

O  O  CL. 


«  E 
4->  ro 
ro  S-  S_ 
CO  Q  O  CO 
4-  O 
4-  S- 
O  CU  Q_ 
O 

CU  S-  • 
O-  3  <C 

>>  o  • 

I —  CO  Or 


38 


medical  area.    It  may  be  that  single  specialty  providers  more  easily  reach 
consensus  on  criteria,  thus  allowing  the  subsequent  activities  (e.g.,  data 
collection,  review,  and  feedback)  to  be  implemented  in  a  more  defined  and 
systematic  manner.    This  finding  may  have  implications  for  the  design  of 
quality  assurance  systems  involving  multi special  ties.    For  example,  systems 
may  be  more  successful  if  topics  and  criteria  are  selected  on  a  departmental 
basis . 

Review  programs  which  were  budgeted  scored  higher  than  those  that 
relied  solely  on  volunteer  support.    It  appears  that  setting  aside  funds 
(whether  from  internal  or  external  sources)  is  important  to  a  site's 
ability  to  score  higher  on  the  ACQAP  Model. 

The  third  category  that  appears  to  be  related  to  high  scores  is  the 
use  of  a  claim  form  as  a  data  source.    Claim  forms  scored  higher  than 
both  chronological  medical  records  and  problem-oriented  medical  records. 
This  finding  in  no  way  implies  that  claim  forms  do  or  do  not  measure  quality 
of  care  better  than  a  chronological  or  a  problem-oriented  medical  record. 
It  does  appear,  however,  that  the  level  and  consistency  of  activity  is 
higher  in  systems  using  a  claim  form  as  a  data  source.    This  finding  might 
be  explained  by  the  fact  that  the  three  settings  that  employed  claim  forms 
work  in  conjunction  with  fiscal  intermediaries.    As  a  result  of  their 
involvement  with  claims  processing,  these  systems  are  well  acquainted  with 
the  use  of  data  and  the  systems  approach  to  program  development  and  manage- 
ment.   Therefore,  when  the  quality  assurance  program  was  developed,  the 
responsible  personnel  may  have  dealt  with  the  task  in  a  more  planned, 
systematic,  documented,  and  specified  manner.    Since  the  systems-oriented 
ACQAP  Model  and  scoring  techniques  measure  such  things  as  extent  of 
planning,  data  availability,  monitoring  of  the  data  system,  and  extent  of 
feedback  (including  payment  denial),  claims  systems  may  have  been  more 
oriented  to  these  characteristics. 

The  interaction  among  the  higher  scoring  categories  is  illustrated 
in  Figures  H,  J,  and  T  of  Appendix  V.    The  major  interaction  which  seems 
apparent  is  that  all  claim  form  data  source  systems  were  budgeted  also. 
Due  to  the  limited  analysis,  it  is  unclear  whether  the  fact  that  the 
claims-based  systems  were  also  budgeted  accounts  for  the  higher  scores. 
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FINDINGS  AND  RECOMMENDATIONS 


Summarized  below  are  the  major  findings  of  the  Ambulatory  Care  Quality 
Assurance  Project  survey.    Findings  focus  on  future  development  and  use  of 
the  ACQAP  Model  and  characteristics  of  programs  as  measured  by  the  Model. 
The  analysis  techniques  employed  do  not  allow  for  predictive  or  probability 
statements  or  for  generalization  beyond  the  sample  of  27  sites.    Thus,  findings 
can  be  interpreted  as  recommendations  to  be  verified  during  program  development 
and  testing. 

In  the  ambulatory  care  quality  assurance  programs  surveyed,  it  appears 

that: 


1.    Application  of  the  ACQAP  Model  substantiated  initial  content 
validity  by  means  of  systematic  comparison  of  model  goals 
and  procedures  in  ambulatory  care  quality  assurance  programs. 
It  was  noted,  however,  that  continued  refinement  of  the  model 
is  needed.    For  example,  the  model  is  not  able  to  determine 
the  existence  of  the  following  elements. 

a.    In  planning  quality  review  activities  most  programs 
anticipated  a  consistent  activity  level,  usually  in 
terms  of  number  of  topics  to  be  reviewed,  amount  of 
data  to  be  collected  and  physician  time  to  be  allo- 
cated to  the  program.    Once  programs  became  opera- 
tional ,  however,  fluctuations  in  the  amount  of  quality 
assurance  activity  usually  occurred  over  long  periods 
(six  to  twenty-four  months).    This  effect  appears  to  be 
directly  related  to  the  service  demands  placed  on  the 
resources  of  the  medical  practice.    Unexpected  fluctua- 
tion in  quality  assurance  activity  was  also  related  to 
the  low  priority  of  quality  assurance  programs  in  many 
of  the  practices.    Quality  review  was  often  viewed  as 
a  function  that  could  be  reduced  or  completely  stopped 
for  a  time  without  having  effect  on  the  quality  of 
services  delivered.    The  current  model  does  not 
measure  this  type  of  activity  or  fluctuations  which 
may  occur. 

o.    Goals  among  programs  varied  considerably.    Cost  and 
utilization  control  was  the  primary  goal  in  some 
programs,  others  established  improved  quality  of 
medical  care  as  the  most  important  goal.    Some  programs 
were  conceived  as  research  projects  with  methodologic 
development  and  improvement  as  the  primary  focus.  Others 
were  interested  in  establishing  an  ongoing  program  as 
part  of  the  operation  of  their  group  practice.  The 
current  model  fails  to  distinguish  among  types  of  goals 
and  there  is  no  mechanism  for  measuring  the  influence  of 
goal  conflict  on  the  success  of  the  program. 
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c.  Realistic  budgeting,  development  of  appropriate  organizational 
structure  and  design  of  effective  management  plans  within  the 
organization  were  important  to  quality  assurance  program 
success.    The  model  is  not  equipped  to  provide  data  needed  to 
make  qualitative  assessment  of  administrative  functions. 

d.  Professional  and  administrative  staff  involvement  and  commit- 
ment in  the  planning  process  and  operational  phases  appear  to 
be  essential  for  program  success.    The  model  does  not  facili- 
tate assessment  of  staff  involvement,  especially  in  the 
planning  phase  where  documented  plans  are  the  only  source  of 
assessment  information. 

e.  Group  practices  with  the  most  success  in  developing  quality 
assurance  programs  were  those  that  had  reached  fiscal  and 
organizational  stability  before  the  review  program  was 
implemented.    The  model  currently  does  not  measure  stability 
of  the  delivery  setting. 

f.  Successful  quality  assurance  program  operations  usually 
assigned  a  person  to  direct  the  program  and  thereby  increased 
staff  commitment  to  the  program  and  consistency  and  continuity 
of  program  activity. 

g.  Many  of  the  quality  assurance  programs  appear  to  be  viewed  as 
projects  rather  than  ongoing  functions  of  the  group  practice. 
This  suggests  that  ambulatory  care  quality  assurance  within 
many  of  the  practices  is  only  at  the  initial  stage  of  evolving 
to  a  routine  function. 

2.  Model  refinement  must  continue  to  focus  on  revision  of  goals,  com- 
ponent names,  and  procedures.    There  appears  to  be  a  need  to  examine 
(a)    word  usage,  clarity,  and  specificity  and  (b)    the  importance  of 
each  procedure  to  a  total  program  and  in  relation  to  other  procedures. 
Empirical  methods,  rather  than  normative,  must  be  developed  to  test 
and  define  the  minimum  set  of  procedures  to  be  included  in  the  model. 

3.  Model  refinement  must  be  aimed  at  developing  methods  for  use  of  the 
model  as  an  assessment  tool ;  to  measure  not  only  whether  a  program 
has  initiated  a  procedure  but  how  well  the  procedure  has  been 
accomplished. 

4.  Currently  application  of  the  ACQAP  Model  has  potential  in  three 
areas;  first  as  a  guide  to  system  design  and  implementation; 
second,  as  a  tool  for  program  assessment  and  third,  to  assist 
in  studies  aimed  at  analyzing  factors  within  programs. 

5.  Volume  II,  "Program  Descriptions"  illustrates  that  a  wide 
diversity  was  to  be  found  across  the  quality  assurance 
programs  surveyed.    The  delivery  settings  and  quality  assurance 
methods  tailored  to  these  settings  varied  substantially.  Cur- 
rently there  is  no  consensus  on  the  most  efficient,  feasible, 
and  productive  quality  review  and  assurance  methods. 
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6.    Program  scores  on  ACQAP  Model  Components  indicate  that: 

a.  There  appears  to  be  a  close  relationship  between  planning 
and  operation  scores  except  for  Component  V  (Identification 
of  Data  Elements  and  Sources).    Across  all  sites  the  average 
Component  V  planning  score  is  0.95  while  the  average  operation 
score  is  only  0.45. 

b.  In  terms  of  component  planning  scores,  Component  IX  (Evaluation, 
Restudy,  and  Further  Action)  and  VIII  (Decision  Making  and 
Feedback)  rank  lowest  with  Component  VII  (Review  Procedures) 
and  Component  II  (Establishment  of  Administrative  Procedures) 
also  scoring  low. 

Operation  scores  indicate  that  Component  IX  and  VIII  again 
are  lowest.    However,  Component  V  (Identification  of  Data 
Elements  and  Sources)  and  Component  IV  (Criteria  Development) 
also  scored  low. 

Across  all  sites  the  components  which  scored  highest  for 
Planning  Score  were  Component  V  (Identification  of  Data 
Elements  and  Scores)  and  Component  III  (Selection  of  Topics 
and  Providers  for  Review).    The  components  which  scored 
highest  on  Operation  Score  were  again  III,  but  also  VI  (Data 
Collection  and  Processing). 

c.  Programs  surveyed  planned  most  in  areas  focusing  on  "Identifi- 
cation of  Data  Elements  and  Sources"  and  "Selection  of  Topics 
and  Providers  for  Review."    "Selection  of  Topics  and  Providers 
for  Review"  was  then  implemented  most  along  with  "Data  Collec- 
tion and  Processing."    However,  although  data  were  being  col- 
lected and  processed,  it  appears  that  lack  of  checks  of  data 
reliability  and  validity  caused  a  low  operation  score  for 
"Identification  of  Data  Elements  and  Sources."  "Criteria 
Development"  also  scored  low. 

Data  collection  and  processing  consume  a  major  part  of  the 
technical  resources  in  many  quality  assurance  programs.  Low 
operation  scores  on  procedures  to  obtain  valid  and  reliable 
information  as  well  as  develop  criteria,  and  relatively  high 
planning  scores  on  data  collection  and  processing  procedures 
suggest  that  emphasis  in  these  areas  should  be  reversed  in 
many  of  the  sites  included  in  the  survey.    If  the  scoring  for 
this  sample  were  to  hold  for  a  larger  representative  sample  of 
ambulatory  quality  assurance  programs  it  would  suggest  that 
developmental  priorities  be  shifted  from  data  collection  and 
processing  to  improvement  of  criteria  development  and  the 
basic  medical  records  and  other  data  sources  used  for  review 
and  assessment  of  the  quality  of  ambulatory  medical  care. 
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7.  Scores  on  ACQAP  Model  procedures  indicate  that: 

a.  Across  all  sites,  on  the  99  procedures  contained  in  the 
model,  only  about  half  the  total  possible  points  were 
scored. 

b.  During  program  planning  and  operation,  special  emphasis 
should  be  placed  on  the  low  scoring  procedures  listed  in 
Figure  11  "Low  Scoring  ACQAP  Model  Procedures"  of  this 
report. 

c.  Many  of  the  low  scoring  procedures  focus  on  determining 
reliability  or  consistency  of  data  accuracy  and  review 
decisions  as  well  as  the  degree  to  which  the  data  and 
review  decisions  are  valid  (i.e.,  based  on  data  that 
accurately  reflect  physician-patient  encounters). 
Feedback  and  restudy  also  contained  many  low  scoring 
procedures.    These  two  components  measure  intervention 
activity  and  resulting  improvement.    The  model  itself 
is  limited  since  it  does  not  directly  measure  cause 
and  effect,  however  it  may  be  hypothesized  from  the 
low  scores  in  procedures  that  one  reason  for  lack  of 
feedback  and  restudy  might  be  lack  of  confidence  in 
the  data  presented  for  review  and,  therefore,  a 
reluctance  to  contact  physicians  and  feed  back  the 
results  of  a  medical  review.    More  attention  to 
determining  data  accuracy  may  increase  likelihood  of 
frequent  feedback  and  restudy  designed  to  measure 
impact  or  change  in  physician  behavior. 

Low  scores  for  the  decision  making  and  feedback  com- 
ponent might  also  be  due  to  an  inherent  reluctance 
to  confront  peer  groups  or  individuals  with  evidence 
of  substandard  care  due  to  lack  of  knowledge  regarding 
effective  intervention  methods  of  modifying  group  and 
individual  behavior. 

8.  Quality  assurance  programs  used  by  groups  of  single  specialty  providers 
(other  than  general  practitioners)  appeared  to  score  high  when  compared 
to  those  of  general  practitioners  and  mul tispecialty  providers.  This 
finding  may  have  implications  for  the  design  of  quality  assurance 
programs  involving  mul tispecial ties.    For  example,  programs  may  be 
more  successful  if  topics  and  criteria  are  selected  on  a  departmental 
basis. 

9.  Quality  assurance  programs  that  were  budgeted  scored  higher  than 
those  that  relied  solely  on  volunteer  support.    It  appears  that 
setting  aside  program  funds  (whether  from  internal  or  external 
sources)  is  important  to  a  site's  ability  to  score  higher  on  the 
ACQAP  model . 
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Presented  above  are  the  major  findings  of  the  ambulatory  care  quality 
assurance  survey.    To  assure  accurate  interpretation,  the  findings  should 
be  read  in  conjunction  with  the  remainder  of  this  volume  and  the  other  two 
volumes  reporting  the  survey. 
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APPENDIX  I 


Ambulatory  Care  Quality  Assurance  Project  Method 

This  appendix  details  the  methods  used  to  collect  the  information 
contained  in  the  project  report.    It  provides  a  step  by  step  description 
of  the  procedures  broken  down  into  seven  specific  tasks: 

A.  Development  of  the  Data  Collection  Instrument  (DCI) 

B.  Survey  of  Potential  Sites 

C.  Participant  Selection 

D.  Data  Collection 

E.  Data  Compilation 

F.  Development  and  Application  of  an  Ambulatory  Care  Quality 
Assurance  Model  to  be  used  in  Assessing  Site  Activities 

G.  Development  and  Application  of  Scoring  Procedures 

The  major  purpose  of  this  investigation  was  to  describe  methods  of 
quality  assurance  review  and  to  develop  major  models  and  strategies  for 
ambulatory  care  quality  assurance  evaluation.    Prior  to  the  development  of 
tasks  A  and  B,  HCMS  staff  held  several  meetings  to  interpret  the  intent  of 
the  contract  and  had  several  exchanges  with  the  Project  Office  at  the 
Bureau  of  Quality  Assurance  (BQA),  Department  of  Health,  Education  and 
Welfare,  to  discuss  specific  directions  the  study  might  take. 


DEVELOPMENT  OF  THE  DATA  COLLECTION  INSTRUMENT  (DCI) 

In  addressing  the  goals  and  objectives  for  this  study,  it  was  first 
necessary  to  identify  areas  where  data  would  have  to  be  collected,  then 
to  determine  the  appropriate  sources  and  types  of  information  for 
investigation.    To  approach  this  task  systematically,  an  open-ended, 
structured  questionnaire  was  developed  from  an  extensive  review  of  the 
current  quality  assurance  literature  and  from  Health  Care  Management 
Systems  (HCMS)  staff  input,  based  on  an  outline  of  project  parameters 
(Appendix  II).    This  first  draft  of  the  DCI  was  designed  to  order  logically 
and  to  classify  data  topics  into  appropriate  categories  for  collection. 
After  two  revisions  based  on  further  in-house  discussions,  the  DCI  was 
forwarded  to  BQA  for  comment.    Following  BQA  input,  the  second  major 
draft  was  produced. 

At  this  stage,  the  DCI  was  pilot  tested  in  an  operational  quality 
assurance  setting  during  an  on-site  visit.    As  a  result  of  the  pilot  test 
and  the  refinement  of  focus,  there  were  two  major  revisions: 

1.    Division  of  the  questionnaire  into  two  components: 

a.  Information  about  the  delivery  systems 

b.  Information  about  the  quality  assurance  program 
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2.    Questions  were  rearranged  to  produce  more  continuity  during  the 
interview  schedule.    The  final  revision  of  the  DCI  was  based 
on  another  intensive  review  of  the  quality  assurance  literature 
to  insure  that  all  applicable  issues  and  questions  were  included. 
A  final  review  of  the  schedule  by  the  project  advisory  board 
was  conducted. 

These  procedures  resulted  in  a  37-page,  226-item,  open-end 
questionnaire  divided  into  two  major  sections,  the  Delivery  System 
(8  pages)  and  the  Quality  Review  Program  (29  pages).    All  parts  of  the 
DCI  were  not  meant  to  be  applicable  to  all  sites.    However,  it  was 
necessary  to  include  all  possible  categories  which  might  contain  data 
on  quality  review  activities.    Essentially,  the  DCI  was  used  as  a  guide  for 
data  collection  and  to  focus  the  direction  of  the  site  interviews.  The 
questions  used  in  the  DCI  are  listed  in  Appendix  II. 


SURVEY  OF  POTENTIAL  SITES 

In  constructing  a  base  population  from  which  a  selected  sample  for 
investigation  might  be  generated,  an  extensive  list  of  sites  engaged  in 
some  form  of  ambulatory  quality  review  activities  was  constructed. 
There  was  no  comprehensive  list  of  sites  involved  in  quality  review,  so 
the  list  came  from  a  variety  of  sources: 

1.  A  list  of  persons  active  in  some  facet  of  quality  assurance  or 
utilization  review,  initiated  and  built  on  through  telephone 
and  correspondence 

2.  Two  major  research  projects  in  quality  assurance  visited  to 
discuss  possible  participants,  on  the  basis  of  their  knowledge 
of  the  field 

3.  A  review  of  quality  assurance  literature  to  create  a  list  of 
ambulatory  quality  assurance  programs  in  operation 

4.  A  review  of  a  31 -page  listing  of  ambulatory  delivery  systems 
having  some  type  of  computer  involvement  in  their  medical  record 
system 

5.  A  review  of  federally  supported  ambulatory  quality  assurance 
programs  supported  by  Social  Rehabilitation  Service,  Bureau  of 
Quality  Assurance,  National  Center  for  Health  Services  Research, 
and  Community  Health  Services 

6.  A  review  of  ambulatory  quality  assurance  programs  supported  by 
private  foundation  funding 

7.  Congressional  hearings  (Competition  in  Health  Services  Market, 
Subcommittee  on  Anti trusts  and  Monopoly,  Committee  of  the 
Judiciary,  U.S.  Senate)  which  listed  approximately  100  ambulatory 
settings 

From  these  sources,  65  sites  involved  in  ongoing  ambulatory  quality 
assurance  activities  were  identified.  All  sites  were  directly  contacted 
with  letters  of  introduction  designed  to  ascertain  the  current  status  of 
their  quality  review  activities,  their  willingness  to  participate  in  the 
study  and  the  suitability  of  their  program  for  a  site  visit  by  the 
research  team. 
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From  these  sources,  65  sites  involved  in  ongoing  ambulatory  quality 
assurance  activities  were  identified.    All  sites  were  directly  contacted 
with  letters  of  introduction  designed  to  ascertain  the  current  status 
of  their  quality  review  activities,  their  willingness  to  participate 
in  the  study  and  the  suitability  of  their  program  for  a  site  visit  by 
the  research  team. 


SITE  SELECTION 

From  the  initial  65  contacts,  a  total  of  45  sites  responded.  Twelve 
of  the  sites  responding  indicated  they  were  unwilling  to  participate  in 
the  study.    Twenty-four  indicated  a  strong  interest;  these  and  nine  others 
who  gave  tentatively  positive  responses  were  contacted  later  by  Tetter  or 
telephone  and  given  a  more  detailed  description  of  the  project.    At  this 
time,  each  site  was  asked  to  send  any  documentation  available  describing 
the  delivery  system,  quality  review  activities  and  other  pertinent 
information. 

After  reviewing  the  information  obtained  from  the  sites,  27  sites 
were  selected  for  possible  site  visits  according  to  the  following  criteria: 

1.  Only  sites  with  ambulatory  quality  review  activities  in  full 
operation  would  be  visited 

2.  Sites  had  to  provide  at  least  some  initial  documentation  about 
the  activities 

3.  Personnel  on  site  should  show  a  high  level  of  interest  and 
willingness  to  participate  in  the  time  consuming  site  visit  and 
data  collection  procedures;  it  was  also  necessary  to  discuss  a 
mutually  convenient  time  for  the  site  visit 

4.  Only  sites  within  the  continental  boundaries  of  the  United 
States  could  be  visited 

5.  Sites  must  be  capable  of  classification  by  the  majority  of 
selected  factors  for  analysis 

The  27  sites  remaining  after  applying  the  above  criteria  were  further 
classified  to  provide  a  listing  of  sites  representative  of: 

a.  Ambulatory  Care  Settings 

b.  Diverse  Geographical  Locations 

c.  Types  of  Quality  Review  Activity 

The  participants  are  listed  in  the  beginning  of  this  volume. 


DATA  COLLECTION 

The  data  collection  procedures  employed  in  this  investigation  involved 
two  separate  but  related  styles.    The  first  of  these  was  the  Pre-Visit 
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230-044  O  -  77  -  5 


Orientation  (PVO).    The  purpose  of  the  PVO  was  to  inform  the  site  visit 
team  of  priorities  for  data  collection,  and  to  give  direction  to  the  actual 
interviews  on  site.    From  this  initial  documentation  it  was  possible  to 
know  approximately  what  information  was  needed  to  complete  the  DCI.  The 
PVO  dealt  with  two  phases  of  the  data  collection  process:    the  selection 
of  a  particular  site  via  the  procedures  described  above,  which  yielded  certain 
descriptive  information,  and  more  complete  documentation  prior  to  the  actual 
site  visit. 

The  PVO  was  designed  to  provide  a  systematic  and  time  saving  approach 
to  the  data  collection  process.    These  pre-visit  activities  can  be  generally 
categorized  as  follows: 

1.  Selection  of  sites  for  possible  inclusion  in  the  study 

2.  Introductory  communication 

3.  Request  for  initial  information  to  determine  participation 
status 

4.  Receipt  of  initial  information 

5.  Request  for  detailed  information  to  initially  compile  DCI 
(in  some  instances,  Section  I  of  the  DCI  was  sent  to  the 
site  for  completion) 

6.  Receipt  of  detailed  information  which  was  subsequently 
recorded  in  the  DCI 

7.  Creation  of  deficiency  list  based  on  information  received 

8.  Request  for  additional  information  (if  available)  and 
arrangement  of  the  site  visit  and  interviews 

These  procedures  varied  somewhat  for  each  site:    some  sites  provided 
extensive  and  comprehensive  documentation  about  both  the  delivery  system 
and  their  quality  assurance  activities,  while  others  were  able  to  provide 
only  general  information  in  these  areas.    As  a  consequence,  the  amount  of 
data  collected  on  site  varied  with  the  amount  of  information  obtained  during 
the  PVO. 

Once  all  PVO  data  were  collected  and  a  site  visit  scheduled,  the 
visit  team  reviewed  all  information  available  on  the  site  prior  to  the 
visit.    Thus,  the  site  team  interviewers  were  well  versed  in  and  familiar 
with  the  documented  operational  procedures  of  the  site  which  facilitated 
interaction  between  interviewers  and  site  personnel  and  helped  to  assure 
accurate  understanding  of  the  site's  program. 

The  series  of  interviews  and  discussions  with  various  personnel  at 
each  site  required  a  day  to  a  day-and-a-half .    Although  it  was  impossible 
to  define  precisely  who  would  be  interviewed  for  all  sites,  it  was  feasible 
to  specify  four  categories  of  persons  to  be  questioned.    In  general,  these 
categories  proved  to  be  sufficient  for  obtaining  information  on  the  quality 
review  programs.    The  categories  were: 

1.  Person(s)  responsible  for  administering  quality  assurance 
auditing 

2.  The  Medical  Director  or  chief  physician 
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3.  Persons  responsible  for  conducting  actual  reviews 

4.  Administrative  personnel  involved  with  quality  assurance 
activities 

The  division  of  labor  for  these  responsibilities  varied  widely  from 
site  to  site. 

The  actual  site  visits  conducted  by  two  senior  research  associates 
followed  a  relatively  consistent  format  for  all  sites,  as  outlined  below: 


Evening  prior  to  visit:    Review  of  notes,  documentation  and  deficiency 

list  for  site. 


Visit:        1.    Meeting  with  site  contact  and  discussion  of  the  schedule 
for  the  day 

2.  Interview  and  discussions  with  person  responsible  for 
the  Quality  Assurance  Program  (QAP)  activities  (approxi- 
mately 1  1/2  hours) 

3.  Interviewing  other  important  QAP  personnel,  committee 
members,  etc.  (1  1/2  hours) 

4.  Interview  with  the  medical  director  (  1  hour) 

5.  Break  for  lunch  (usually  involved  informal  discussion 
with  QAP  personnel) 

6.  Review  of  information  collected  during  the  morning  and 
creation  of  a  second  deficiency  list  to  give  focus  to  the 
afternoon  interview  (1  hour) 

7.  Interview  with  personnel  actually  doing  quality  review 
(approximately  1  1/2  hours) 

8.  Interview  with  clinic  administrator  (1  hour) 

9.  Concluding  interviews  with  personnel  responsible  for 
QAP  activities  (approximately  45  minutes) 


An  attempt  was  made  to  tape  record  all  interviews  for  later  transcription 
and  review.    However,  some  persons  declined  to  be  recorded,  and  in  some 
situations  the  surroundings  did  not  permit  use  of  a  recorder.    After  reviewing 
primary  source  documentation,  investigators  would  interview  participant  per- 
sonnel to  collect  information  on  the  operational  status  of  the  quality 
assurance  activities.    The  other  major  activity  of  the  on-site  investigation 
was  to  review  substantiating  evidence  for  a  participant's  quality  assurance 
program.    The  team  usually  asked  to  see  some  data,  e.g.,  minutes  from 
meetings,  to  support  verbal  statements  about  a  program.    Time  constraints 
and  other  factors  made  it  impossible  to  conduct  all  interviews  separately, 
as  the  investigators  had  requested.    In  a  situation  where  more  than  one 
person  was  interviewed,  tape  recording  also  proved  unfeasible.    In  all 
interviews,  both  interviewers  took  extensive  notes.    On  the  average,  five 
people  per  site  were  interviewed. 

When  one  day  was  insufficient  time  to  gather  all  pertinent  information, 
the  site  visit  team  returned  the  following  day,  if  possible.    Usually  these 
second-day  visits  were  used  to  review  committee  minutes,  to  speak  with 
additional  ancillary  personnel  and  to  follow  up  on  specific  questions  not 
covered  during  the  previous  day. 
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DATA  COMPILATION 


Immediately  after  the  conclusion  of  a  site  visit,  data  compilation 
began.    Both  site  visitors  reviewed  and  dictated  their  notes  separately. 
Along  with  documentation  provided  by  the  site  and  the  initial  information 
from  the  DC  I,  these  notes  were  placed  in  a  reference  file  for  the  site. 
At  this  point,  the  reference  file  was  reviewed  for  completeness  and 
necessary  follow-up  information  was  collected. 

Return  visits  to  three  sites  were  necessary  for  further  data  collection 
although  most  of  the  follow-up  information  was  collected  by  telephone  or 
letter.    Once  a  file  was  judged  complete,  it  was  summarized  into  a  single 
narrative  description  of  the  site. 

The  process  of  generating  the  descriptions  for  this  portion  of  the 
study  involved  many  steps  in  addition  to  those  outlined  above.    Each  site 
presented  its  own  peculiarities  during  the  writing  of  the  description.  The 
steps  involved  usually  included  most  of  the  following: 

1.  Review  of  transcribed  tapes,  notes  and  the  DCI 

2.  Listening  again  to  taped  interviews 

3.  Reviewing  site  grant  applications 

4.  Reviewing  committee  minutes 

5.  Contacting  the  site  again 

6.  Extensive  discussion  between  site  visit  investigators 

Each  program  description,  written  in  accordance  with  a  prescribed  format 
to  assure  uniformity  for  all  site  descriptions,  was  prepared  by  a  staff 
member  and  subjected  to  the  three-level  review  process  outlined  below: 

Review  I      Content  review  of  initial  draft 

Review  II     Format  review  of  second  draft 

Review  III    Final  review  by  senior  research  associate 

Thus  the  final  reference  file  for  each  site  includes: 

1.  A  narrative  description  of  the  program  including  the  delivery 
system  and  the  quality  review  program  (See  Volume  II:  Program 
Descriptions) 

2.  All  source  documentation  provided  by  the  site 

3.  Transcribed  tapes 

4.  Site  visit  notes 

5.  The  DCI 

6.  Any  other  relevant  material 

a.  Articles 

b.  Grant 

c.  Minutes 

DEVELOPMENT  AND  APPLICATION  OF  THE  ACQAP  MODEL 

The  development,  application,  and  scoring  of  the  Ambulatory  Care 
Quality  Assurance  Model  is  described  in  the  body  of  this  volume. 
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CLASSIFICATION  OF  SURVEY  SITES  BY  FACTORS  1-6 


Twenty-three  of  the  twenty-seven  survey  sites  were  used  in  the 
analysis  of  Factors  1-6;  three  sites  were  excluded  because  they  were 
classified  as  research  programs  and  one  was  excluded  due  to  lack  of 
data  at  the  time  of  analysis.    The  chart  below  describes  reasons: 


Factor 

No.  of  Sites 
included  in 
Mna lysi s 

No.  of  Sites 
excluded 
Trom  analysis 

Reasons  for 
Excluding  Sites 
oy  factor 

1. 

Type  of  Group 
Practice 

17 

6 

Two  sites  did  not 
deliver  medical 
services  and  four 
sites  were  involved 
in  multiple 
facilities  rather 
than  a  single 
facility. 

2. 

Type  of  Revenue 
Source  to 
Delivery  System 

18 

5 

Three  sites  did  not 
deliver  medical 
services  and  two 
sites  had  multiple 
revenue  sources 
which  did  not  match 
the  categories. 

3. 

Length  of 
Program 
Activity 

23 

0 

4. 

Type  of  Q.A. 
Program 
Support 

23 

0 

5. 

Type  of  Q.A. 
Review 

22 

1 

One  site  used  a 
combination  of 
external  and  internal 
review. 

6. 

Type  of  Data 
Source  for  Q.A. 
Program 

20 

3 

One  site  used  multiple 
facilities  with  a 
mixture  of  data 
sources  and  two  sites 

used  a  computerized 
data  format. 
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APPENDIX  II 
Data  Collection  Instrument  (DCI) 


SECTION 

1:    Delivery  Svstem 

SECTION 

2:    Qualitv  Review  Proaram  (ORP) 

Part 

i  • 

Identification  Sheet 

Part 

1 

Identification  Sheet 

Part 

rui  u 

2 

Current  Structure 

Part 

2: 

Current  Structure 

Part 

3 

Services  and  GeoaraDhic  Factors 

Part 

4 

Uti 1 ization 

Part 

3: 

Scope  of  Services 

Part 

5 

Medical  Record 

Part 

6 

Billing 

Part 

4: 

Fiscal  Factors 

Part 

7 

Abstracting 

Part 

8 

Data  Analysis 

Part 

5: 
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The  questions  used  in  the  Data  Collection  Instrument  are  listed  in  the 
following  pages.    In  the  original  DCI,  ample  space  was  allowed  for  answers 
and  comments. 
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SECTION  1 :    DELIVERY  SYSTEM 


Part  1:    Identification  Sheet 

1 .  Site  name 

2.  Address 

3.  Telephone  number 

4.  Date  visited 

5.  Date  scheduled  for  visit 

6.  Administrator 

7.  Medical  director 

8.  Staff  contact  for  HCMS 

9.  Starting  date  of  operation: 

10.  Clinic 

11.  Quality  review  program  (QRP) 

12.  Type  of  QRP  planned 

13.  Comments: 


Part  2:    Current  Structure 

14.  We  would  like  to  have,  in  any  form,  a  description  of  the  physical  plant 
(including  information  on  the  following): 

Number  and  description  of  equipment 

Maximum  allowed  in  facility 

Number  of  waiting  areas 

Number  of  examination  rooms 

Number  of  physician  offices 

Ancillary  service  space  (square  footage) 

15.  Could  you  provide  an  organizational  chart  giving  details  on  the  following 
positions: 

Administrative 
Medical 

Board  of  Trustees 

Departments 

Committees 

16.  Could  you  provide  FTE  information  on  your  personnel  positions: 

Administrative 
Medical 

17.  Briefly  describe  type  of  organization  of  medical  staff 

18.  Describe  ownership  of  clinic  or  office 

19.  Describe  accreditation 

20.  Describe  affiliations  (e.g.,  professional  groups,  teaching  university,  etc.) 

21.  Describe  relationship  to: 

State  or  local  medical  association 
PSRO 

Governmental  agencies 
Licensing  boards 
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Part  3:    Scope  of  Services 


22.  Please  provide  documentation  of  the  services  offered.    If  not  available, 
could  you  list  general  categories  of  services? 

23.  What  specific  benefit  packages  does  your  clinic  or  office  offer?  Please 
provide  documentation  of  such  packages. 

24.  Provide  a  description  of  population  served  (social  and  demographic 
characteristics) . 

25.  Are  geographic  factors  a  concern  for  your  clinic?  If  so,  do  you  know  the 
farthest  an  individual  travels  to  receive  care?  Do  you  know  the  farthest 
an  individual  travels  for  referral  services? 

26.  Please  describe  unique  geographic  factors  and  whether  or  not  you  feel  they 
affect  the  accessibility  of  care  for  your  population. 

27.  Comments: 


Part  4:    Fiscal  Factors 

28.  Describe  funding  sources  which  support  the  delivery  of  care. 

29.  In  what  form  does  clinic  or  plan  receive  funding  (indicate  %): 

fee-for-service 

insurance 

prepaid 

Medicaid 

Medicare 

30.  What  is  your  total  funding?    Indicate  dates  of  funding. 

31.  What  were  initial  capital  investments  in  your  clinic  or  offices? 

32.  Could  you  provide  cost  figures  for  staffing  (medical,  administrative)? 

33.  Could  you  provide  total  operational  cost  (separate  cost  components,  if 
available)? 

34.  Comments: 


Part  5:    Utilization  Patterns 

35.  Could  you  provide  documentation  for  the  following  areas: 

number  of  enroll ees  or  members  of  plan  or  clinic 

number  of  encounters  per  year 

utilization  of  enrol  lees/members  by  service 

36.  Time  of  day  when  most  service  is  delivered  (  indicate  %): 

a.m. 
p.m. 

37.  Can  you  provide  documentation  on  utilization  patterns  by  department? 

38.  Can  you  provide  documentation  on  utilization  patterns  by  diagnosis? 

39.  Can  you  provide  documentation  on  utilization  patterns  by  procedure? 

40.  Comments: 


Part  6:    Medical  Record 

41.    Could  you  provide  documentation  or  describe  medical  record  system 
procedures  (specific  data  flow  characteristics)? 
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42.  Could  you  provide  a  medical  record  copy  or  a  list  of  data  elements? 

Data  elements 

ID  sheet/coverage 

Patient  ID  (coding  system) 
Provider  ID  (coding  system) 
Clinical  data 

Diagnosis,  treatment,  follow-up 
Lab  sheet/X-ray 
Ancillary  services 
Drugs 

Consul tation/referral s 

43.  Could  you  classify  the  medical  record  as  one  of  the  following  (circle): 

POMR 

Chronological 
Source  oriented 
Time  oriented 
Other: 

44.  Is  data  recorded  in  other  places  (e.g.,  nurses  station  card  file)  besides 
chart  for: 

Diagnostic  procedures 
Treatment  procedures 
Follow-up  procedures 
If  so,  please  describe. 

45.  Comments: 


Part  7:  Billing 

Note:    If  copy  of  billing  form  supplies  answers  to  following  question,  please 
attach  billing  form  and  comments. 

46.  Could  you  describe  your  billing  process  (briefly)? 

47.  At  what  point  are  services  for  one  encounter  noted  for  billing  purposes? 

48.  What  (if  any)  codes  are  used  to  describe  various  diagnostic  and  treatment 
procedures?  Describe. 

49.  What  data  elements  are  included  on  billing  forms? 

50.  The  number  of  personnel  involved  with  billing  process 

51.  Comments: 


SECTION  2:    QUALITY  REVIEW  PROGRAM  (QRP) 


Part  1:    Identification  Sheet 

1.  Why  was  QRP  initiated? 

2.  What  are  the  stated  goals  and  objectives  of  your  QRP? 

3.  How  was  agreement  reached  on  goals  and  objectives? 

4.  How  was  your  QRP  approach  selected?    Describe  the  rationale. 

5.  Was  it  necessary  to  define  "quality"  in  your  program?    If  so,  how  did 
you  do  so? 

6.  Describe  amount  of  literature  research  done  prior  to  quality  assurance 
program. 
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7.  Describe  the  use  of  consultants  to  develop  and  implement  quality  assurance 
program. 

8.  How  would  you  describe  your  attitude  toward  assessing  quality  of  care? 

9.  What  unique  characteristics  of  the  practice  setting  affected  the  development 
and  implementation  of  your  quality  review  program?    (e.g.,  turnover  rate  of 
physicians,  patient  population,  historical  development  of  medical  staff  or 
cl inic) 

10.    Describe  any  special  problems  or  considerations  which  affected  the 
implementation  of  quality  review. 


Part  2:    Current  Structure 

11.  Could  you  provide  an  organizational  chart  giving  details  on  the  following 
positions  in  the  QRP: 

administrative 

clerical 

medical 

committees 

departments 

Board  of  Trustees  role 

12.  Could  you  provide  numerical  information  on  the  following  personnel 
positions  in  QRP: 

administrative 
medical 
specialists 
consul tants 
technologists 

13.  Describe  duties  of  personnel  in  QRP.    Attach  job  descriptions  if  possible. 

14.  Describe  input  received  by  outside  groups  or  affiliations  for  QRP  (e.g., 
teaching  universities,  medical  associations,  etc.). 


Part  3:    Services  and  Geographic  Factors 

15.  Provide  documentation  on  the  services  (either  diagnostic  or  treatment) 
reviewed  in  QRP. 

16.  How  were  services  chosen  for  review? 

17.  Describe  (by  practice  type)  the  providers  reviewed  in  the  QRP. 

18.  Are  geographic  factors  a  concern  in  QRP?    If  so,  please  describe. 


Part  4:  Utilization 

19.  Are  utilization  figures  used  in  QRP?    If  so,  how  are  they  employed  (e.g., 
frequency  distributions,  statistical  variations,  etc.)? 

20.  Do  you  specifically  measure  for  over-  and/or  under-utilization  in  QRP? 
If  so,  please  describe. 

21.  Describe  the  number  of  encounters  reviewed  in  QRP. 
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Part  5:    Medical  Record 


22.  Are  medical  records  used  in  QRP?    If  so,  please  describe  how. 

23.  What  medical  record  data  elements  are  used  in  QRP? 

24.  How  many  people  in  the  medical  records  department  are  concerned  solely 
with  QRP? 

25.  How  many  personnel? 

26.  Do  QRP  medical  records  personnel  have  special  training?    Or  duties? 

27.  Are  diagnostic  and  treatment  procedures  medically  verified  for  purposes 
of  quality  review? 

28.  Where  does  medical  record  go  for  QRP  processing? 

29.  Are  special  forms  (e.g.,  encounter  forms)  included  in  charts  for  strictly 
QRP  purposes?    Please  attach. 


Part  6:  Billing 

30.  Are  billing  forms  used  in  the  QRP?    If  so,  please  describe. 

31.  What  data  elements  of  billing  form  are  used? 

32.  Who  collects  billing  data  elements? 

33.  How  many  personnel  are  involved? 

34.  Do  personnel  have  any  special  duties  or  training  for  QRP  data  collection? 

35.  Where  does  billing  information  go  for  the  QRP? 

36.  If  billing  is  used  in  QRP,  are  procedures  employed  to  affect  medical 
performance  (e.g.,  payment  denial)?    If  so,  please  describe. 


Part  7:  Abstracting 

37.  Is  abstracting  done  for  QRP  purposes?    If  so,  please  describe. 

38.  Could  you  please  provide  a  copy  of  the  abstract  form? 

39.  What  data  elements  are  abstracted  and  from  what  source? 

40.  Is  coding  used  to  transfer  information  to  different  format?    If  so,  what 
codes  are  used? 

41.  Are  explicit  guidelines  developed  for  abstracting?    Please  attach. 

42.  What  personnel  are  involved  in  abstracting?    How  many?    How  are  people 
trained  to  abstract? 

43.  Is  abstracting  procedure  monitored?    If  so,  describe. 

44.  What  is  reliability  of  abstract  monitoring  system? 

45.  How  confident  are  you  about  the  reliability  of  your  abstracting  process? 

46.  What  is  an  acceptable  error  rate? 

47.  Are  reliability  studies  available?    If  so,  please  provide. 

48.  Who  receives  copies  of  abstracts? 

49.  Is  abstract  data  verified  by  medical  personnel  prior  to  review? 

50.  Is  abstract  processed  manually  or  computerized? 


Part  8:    Data  Analysis 

51.    Describe  what  data  base  is  analyzed  for  purposes  of  QRP. 
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52.  Are  QRP  data  elements  analyzed  manually  or  ADP?    If  ADP,  also  answer 
questions  in  Part  9. 

53.  What  types  of  aggregate  data  are  available  for  analysis? 

utilization  figures 
diagnostic  categories 
treatment  categories 
services 

54.  How  often  is  data  base  updated  for  use  in  QRP? 

55.  What  types  of  ongoing  analysis  are  done  in  QRP? 

56.  What  types  of  special  analysis  are  done  in  QRP? 

57.  What  personnel  are  involved  in  analysis  of  data  for  QRP?    What  are  their 
skill  levels? 

58.  What  reports  are  made  available  to: 

administrative  staff 
medical  staff 
others 

59.  What  is  the  format  of  such  reports? 

60.  Are  individual  patient  profiles  available  for  QRP  analysis? 

61.  Are  characteristics  of  QRP  data  presented  in: 

numbers 

frequency  distributions 

means 

percent 

percentiles 

62.  Which  data  array  seems  most  effective  in  QRP  or  which  one  is  most 
accepted  for  QRP? 


Part  9:    Computer/Automated  Data  Processing 

63.  Could  you  describe  the  use  of  automated/computerized  data  processing  in 
your  cl inic?    Describe  any  company  with  which  you  work. 

64.  Could  you  describe  the  use  of  automated/computerized  data  processing  in 
your  QRP? 

65.  Describe  the  reasons  for  implementing  an  automated  system  in  your  setting. 

66.  How  long  has  your  automated  system  been  operational? 

67.  How  many  personnel  are  involved  in  the  operation  of  the  automated  system? 

68.  Are  computer  programs  used  in  QRP  (e.g.,  Biomed,  SPSS,  specially  developed 
programs)? 

69.  Describe  the  use  of  hardware. 

70.  Do  you  lease  or  own  hardware?    Do  you  use  hardware  in  some  other  facility 
(e.g. ,  university)? 

71.  Are  computerized  criteria  standards  used  to  evaluate  quality?    If  so, 

how  are  they  entered  into  computer?    Describe  the  type  of  program  developed. 

72.  How  is  data  entered  into  automated  system? 

73.  What  types  of  coding  are  done  prior  to  entering  data  into  system? 

74.  What  is  turnaround  time  in  the  automated  system? 

75.  Describe  the  major  problems  in  software,  hardware,  and  implementation  of 
the  total  automated  system. 
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Part  10:    Case  Selection  Process 


76.  In  QRP,  are  cases  selected  for  review  or  evaluation?    If  so,  what  criteria 
are  employed  for  selection?    Please  describe. 

77.  Describe  sampling  procedures  to  select  cases. 

78.  Who  selects  the  cases  for  review? 

79.  What  types  of  data  are  taken  from  these  selected  cases? 

80.  What  uses  are  made  of  cases  or  data  taken  from  such  cases?  Describe. 

81.  Who  has  access  for  review  purposes  to  selected  cases?    Do  they  have  access 
to  other  data  regarding  cases? 

82.  Are  the  same  personnel  who  review  the  only  persons  who  see  data  from 
selected  cases? 

83.  If  criteria  are  established  to  evaluate  cases,  how  does  examiner  determine 
variance? 

84.  What  happens  to  variant  cases  after  review?    How  is  that  information  processed 
for  QRP  purposes? 

85.  What  is  the  time  frame  involved  from  case  selection  to  review? 

86.  What  does  QRP  do  to  examine  cause  of  deviances? 

87.  Is  the  purpose  of  case  selection  to  identify  individual  error  or  to  discover 
patterns  that  require  change? 


Part  11:    Special  Studies 

88.  Does  QRP  include  the  use  of  special  studies?  Describe. 

89.  If  so,  how  are  they  oriented  to  solve  specific  problems? 

90.  How  are  topics  selected  for  special  study? 

91.  Are  samples  employed  to  collect  information  for  special  studies? 

92.  How  is  sample  determined  (e.g.,  random  cases,  high  risk  groups,  age  groups)? 

93.  What  types  of  study  formats  are  used  in  special  studies  (e.g.,  observational, 
clinical  trial,  pre-post  design)? 

94.  Are  statistical  display  methods  used  for  identification  of  deviance? 

95.  If  special  studies  address  a  specific  diagnosis  or  treatment,  how  are  those 
cases  "flagged"  from  the  population? 

96.  How  do  you  follow  up  on  special  studies  to  measure  impact? 


Part  12:  Criteria/Standards 

97.  Are  criteria/standards  employed  for  quality  review  purposes? 

98.  How  were  they  developed? 

99.  Who  developed? 

100.  Were  they  adopted  from  other  programs? 

101.  Were  they  developed  inhouse? 

102.  Was  there  broad  inhouse  consensus  on  the  criteria/standards  used  in  QRP? 
Please  describe  the  procedure  for  such  consensus. 

103.  Are  the  same  personnel  who  developed  the  criteria  evaluated  by  the  same 
criteria? 

104.  Were  criteria/standards  developed  from  academic  research  and  standards 
(e.g.,  literature,  expert)  or  were  they  developed  based  on  empirical 
evidence? 
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105.  Describe  the  administrative  implementation  of  the  criteria  and  standards. 

106.  Describe  the  role  criteria/standards  play  in  assessing  total  quality  of 
service  delivered  in  the  clinic. 

107.  What  evidence  is  available  which  indicates  the  effectiveness,  acceptability, 
and  accuracy  of  your  criteria/standards? 

108.  If  criteria/standards  are  available,  could  you  please  provide  a  copy. 

109.  Are  the  criteria/ standards  implicit  or  explicit? 

110.  How  was  the  stringency  of  standards/criteria  determined? 

111.  How  are  criteria/standards  updated  and  reformulated?    Please  describe. 


Part  13:    Process  and  Outcome 

112.  Assuming  that  criteria/standards  are  employed  in  the  QRP: 

Does  criteria  focus  on  process,  outcome,  or  both? 
How  are  criteria/standards  related  to  processes  of  care? 
How  are  criteria/standards  related  to  different  end  results 
or  outcomes? 

113.  Assuming  the  distinction  between  process  and  outcome  criteria,  in  your 
opinion  which  is  most  important  for  quality  review?  Describe  how  your 
QRP  deals  with  this  issue. 

114.  Was  literature  research  for  use  in  QRP  cited  to  develop  emphasis  on 
either  process  or  outcome  measures? 

115.  Are  different  criteria  and  standards  developed  for  different  patient 
population  or  practice  settings? 


Part  14:    Critical  Variables  (Weighting) 

116.  Does  your  QRP  consider  certain  variables  (e.g.,  procedures)  of  quality 
review  more  important  than  others? 

117.  If  so,  are  weighting  or  indexing  schemes  employed  in  QRP  to  emphasize 
more  critical  variables? 

118.  What  variables  are  weighted? 

119.  Describe  processes  employed  (e.g.,  factor  analysis)  to  weight  such 
variables. 

120.  How  was  the  weighting  scheme  determined? 

121.  Describe  impact  of  weighting  scheme  on  quality  review. 


Part  15:    Review  Procedures 

122.  Are  review  procedures  employed  in  your  QRP?    Please  describe  the  type  of 
review  done. 

123.  How  was  the  review  procedure  developed?  Describe. 

124.  What  materials  are  present  at  review  sessions?   Who  arranges  materials 
and  schedules? 

125.  If  medical  records  are  reviewed,  is  the  record  reviewed  directly,  through 
abstract,  encounter  forms,  and/or  claim  forms? 

126.  How  many  personnel  are  involved  in  the  review  process?    Note  whether  they 
are  medical  or  administrative  personnel. 
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127.  Are  the  data  used  in  the  review  always  complete?    If  not,  what  is  used? 

128.  Is  record  data  verified  prior  to  review? 

129.  If  medical  personnel  review,  how  is  their  judgment  validated  for  accuracy? 

130.  How  do  you  control  bias  or  external  influences  on  the  review  process? 

131.  How  is  reliability  of  review  determined?  Describe. 

132.  Are  reviews  compared  with  other  reviews? 

133.  Is  there  any  evidence  of  a  relationship  between  recording  procedures  and 
the  review  records  to  quality  review? 

134.  Is  the  review  linked  to  corrective  action?  Describe. 

135.  How  are  reviewers  trained  (either  clerical  or  medical)?  Describe. 

136.  Is  randomization  used  for  assigning  cases  to  reviewers?    If  not,  describe 
how  cases  are  distributed  to  reviewers. 

137.  Are  there  specialized  guidelines  established  to  govern  review  procedures? 
Please  attach. 


Part  16:    Episode  of  Illness 

138.  Does  QRP  evaluate  episodic  patient  illnesses?    If  so,  how  is  an  episode 
of  illness  defined? 

139.  How  are  episodes  chosen? 

140.  Are  patient  profiles  dealing  with  specific  episodes  available? 

141.  Are  provider  based  profiles  available  for  review  dealing  with  specific 
episodes? 

142.  How  are  data  elements  of  care  linked  in  a  specific  time  frame  for  review? 


Part  17:    Feedback  Action 

143.  Does  a  system  for  feedback  to  providers  and/or  patients  exist  in  your  QRP? 

144.  Describe  the  system  of  feedback  based  on  your  quality  review? 

145.  What  is  the  corrective  action  process  (e.g.,  strictly  administrative, 
strictly  educational,  etc.)? 

146.  Is  medical  education  of  staff  an  important  feedback  mechanism  in  your  QRP? 
Describe  your  medical  education  program. 

147.  If  so,  how  are  these  educational  needs  determined? 

148.  In  terms  of  educational  feedback,  are  there  links  with  educational 
institutions  (e.g.,  university  teaching  hospitals,  medical  associations, 
etc.)? 

149.  How  is  corrective  action  implemented?    Who  initiates  such  action? 

150.  Describe  the  procedures  by  which  a  provider  can  appeal  or  question 
feedback. 


Part  18:    Impact  Studies 

151.  Have  you  done  any  impact  studies  to  determine  the  effects  of  your  quality 
review  activities?    If  so,  could  you  please  provide  results? 

152.  What  components  of  your  QRP  were  measured  or  evaluated? 

153.  Was  any  evaluation  of  consumer  satisfaction  done?    Of  provider  satisfaction? 

154.  If  education  is  a  part  of  your  QRP,  did  you  evaluate  the  impact  of  your 
educational  program? 
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Part  19:    Consumer  Satisfaction 


155.  Are  patients  questioned,  in  any  form,  about  the  quality  of  medical  care 
delivered  in  your  setting? 

156.  If  so,  how  are  these  patients  questioned? 

157.  Do  you  measure  any  acceptability  variables?    (For  example,  e.g.,  appointment 
time,  waiting  time,  patient-provider  relationships,  etc.) 


Part  20:    General  Attitudes 

158.  Do  you  feel  the  quality  of  medical  care  in  your  clinic  is: 

1.  Excellent 

2.  Good 

3.  Fairly  Good 

4.  Adequate 

5.  Poor 

159.  Do  you  feel  the  competency  of  the  medical  staff  is: 

1.  Excellent 

2.  Good 

3.  Fairly  Good 

4.  Adequate 

5.  Poor 

160.  Could  you  describe  five  variables  which  affect  the  quality  of  care  within 
your  setting? 

1. 
2. 
3. 
4. 
5. 

161.  Is  it  worthwhile  to  assess  the  performance  of  medical  providers? 

Yes  No 

Explain  answer. 


Part  21:    Fiscal /Cost  Factors 

162.  What  funding  sources  support  your  QRP  (e.g.,  research  grants,  federal 
subsidies,  etc.)? 

163.  What  is  the  total  budget  for  QRP?    Over  what  time  period? 

164.  Is  QRP  funding  related  to  third  party  payers  (e.g.,  percentage  of  billing 
cost  saved  and  channeled  to  QRP)? 

165.  Do  you  have  figures  or  estimates  on  QRP  expenditures  per  patient?  Per 
physician?   Per  administrative  staff? 

166.  Over  a  specific  period  of  time  (e.g.,  last  six  months),  what  has  been  the 
QRP  cost  trend? 

167.  What  are  the  QRP  costs: 

Medical  staff  cost  - 
Administrative  staff  - 
Clerical  staff  - 

Other  cost  categories  (please  be  as  specific  as  possible)  - 

168.  Was  any  initial  capital-type  investment  made  for  QRP  (e.g.,  computer 
system,  new  offices,  special  forms,  etc.)? 
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Costs  for  Computer/Automated  Data  System: 


1.  What  was  the  total  developmental  cost  for  the  computer/automated  system? 

2.  What  is  the  total  operational  cost  of  the  computer/automated  data  system? 

3.  What  are  the  funding  sources  for  these  costs? 

4.  How  many  personnel  are  involved  in  operating  computer/automated  system 
for  QRP: 

Programmers  - 
Analysts  - 
Consultants  - 
Data  Clerks  - 
Coders  - 

Computer  Operations  - 

5.  Please  note  personnel  cost  (both  FTE  and  PTE). 

6.  What  were  the  equipment  costs  for  use  in  QRP  (e.g.,  keypunch  machine, 
CRT  devices,  etc.)? 

7.  The  cost  of  supplies  for  the  computer/automated  system  for  QRP? 
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APPENDIX  III 

Ambulatory  Care  Quality  Assurance  Project  Model  Consultants 


1.  Dr.  Robert  Brook,  University  of  California  School  of  Medicine,  Los  Angeles, 
and  The  Rand  Corporation,  Santa  Monica,  California 

2.  Dr.  John  Collette,  Associate  Professor,  Department  of  Sociology,  University 
of  Utah,  Salt  Lake  City,  Utah 

3.  Dr.  Joseph  S.  Gonnella,  Director  of  Medical  Education,  Jefferson  Medical 
College,  Philadelphia,  Pennsylvania 

4.  Dr.  Robert  Kane,  Department  of  Family  and  Community  Medicine,  University 
of  Utah,  College  of  Medicine,  Salt  Lake  City,  Utah 

5.  Dr.  James  P.  LoGerfo,  Department  of  Health  Services,  School  of  Public 
Health  and  Community  Medicine,  University  of  Washington,  Seattle,  Washington 

6.  Jeanne  Magagna-Deuschle,  Deputy  Director,  Peer  Review  Program,  Health 
Insurance  Plan  of  Greater  New  York,  New  York 

7.  Marie  Michnich,  Ambulatory  Care  Evaluation  Project,  School  of  Public 
Health,  University  of  California,  Los  Angeles,  California 

8.  Dr.  Beverly  C.  Payne,  Office  of  the  Deans,  University  of  Michigan,  School 
of  Medicine,  Ann  Arbor,  Michigan 

9.  Dr.  James  Roberts,  The  Bureau  of  Quality  Assurance,  Department  of  Health, 
Education,  and  Welfare,  Rockville,  Maryland 

10.  Dr.  Nora  Lou  Roos,  University  of  Manitoba,  Faculty  of  Medicine,  Department 
of  Social  and  Preventive  Medicine,  Winnipeg,  Canada 

11.  Dr.  Paul  J.  Sanazaro,  University  of  California  School  of  Medicine, 
San  Francisco,  California 

12.  Dr.  John  W.  Williamson,  Department  of  Health  Care  Organization,  Johns 
Hopkins  University,  School  of  Hygiene  and  Public  Health,  Baltimore,  Maryland 

13.  Eugene  C.  Wood,  Director  of  Ambulatory  Care,  Department  of  Health  Planning 
and  Delivery,  American  Hospital  Association,  Chicago,  Illinois 

14.  Dr.  George  Whitworth,  Department  of  Peer  Review  Coordination  and  Appeals, 
California  Medical  Association,  San  Francisco,  California 

15.  Dr.  Mark  Pearlman,  Community  Health  Services,  Bureau  of  Health  Services 
Research,  Department  of  Health,  Education,  and  Welfare,  Rockville,  Maryland 
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APPENDIX  IV 


Site 

Interrater 
Judges  1  &  2 

Rel iabi 1 ities 

Judge  3 
With  1       With  2 

Overal 1 
Average 

,                         ,  ad.  

01 

.88 

.84 

.79 

.84 

02 

.92 

.69 

.68 

.76 

03 

.96 

.78 

.79 

.84 

04 

.94 

05 

.93 

06 

.90 

.80 

.75 

.82 

07 

.96 

.74 

.73 

.81 

08 

.97 

.68 

.69 

.78 

09 

.97 

10 

.89 

11 

.93 

.76 

.75 

.81 

12 

.91 

13 

.90 

14 

.98 

.81 

.81 

.87 

15 

.93 

16 

.97 

.77 

.76 

.33 

17 

.83 

18 

.93 

.78 

.76 

.82 

19 

.96 

20 

.88 

21 

.99 

.85 

.85 

.90 

22 

.94 

23 

.93 

24 

.94 

.71 

.73 

.79 

Overall  Average  Reliability  -  87% 

Judges  1  and  2  were  the  site  investigators. 

Judge  3  was  another  HCMS  staff  member. 
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APPENDIX  V 
FIGURES:    SCORES  BY  SETTING  AND  QUALITY 
ASSURANCE  PROJECT  CHARACTERISTICS 
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